detached 75fae3a

feat(broker): add atomic host store migration

commit
75fae3a98302d58e3829935103ec057fc5390b83
author
brice · · 2026-09-16 12:04 UTC
parent
a3b975e
signature
GPG VerifiedCD88F9E564E0102646FB6077A1A07C7C7C3E349D · arch-gpg Verified for the committer email
22 files · +3159 -143
Unified
UnifiedSplit
A internal/adapters/brokerstore/README.md +89 -0
@@ -0,0 +1,89 @@
1+# Offline broker store (Plan 001 P2.3)
2+
3+`OpenOffline` is deliberately not wired into application composition. Supply a
4+private destination directory, immutable copies of any legacy inputs, and an
5+explicit policy for each imported endpoint. An empty input path means no input;
6+a named missing, empty, corrupt, or unsupported file fails closed. The lifetime
7+lock is `hosts.json.lock`, the exact lock file the legacy host writer takes, so
8+a store opened on a directory that live writer also uses excludes it for the
9+lifetime of the store. The legacy cache source has no such lock — its writer
10+takes none — so a cache source must be immutable or quiesced before migration.
11+Never point migration at a cache file that a writer can still mutate.
12+
13+Each source's recovery manifest entry records its generic role label
14+(`membership` for the host registry, `observations` for the catalogue cache),
15+whether the source was supplied at all, and the SHA-256 of its bytes. Presence
16+is separate from the digest because an absent source and an empty one can share
17+a digest; a record that claims a source was absent while retaining its bytes is
18+rejected.
19+
20+## Transaction and recovery
21+
22+- Hold `hosts.json.lock` with Linux nonblocking exclusive `flock` until `Close`.
23+ The name matches the legacy host writer's lock, so the store and that writer
24+ exclude each other; the legacy catalog cache writer takes no lock at all.
25+- Validate bounded legacy hosts v1/v2/v3 and cache v2/v3/v4 without calling their
26+ existing writer adapters. Cache decoding freezes the existing exact legacy
27+ DTOs here; count-only v2 inventories are deliberately unsupported.
28+- Write `recovery.json`: original bytes, SHA-256 migration manifest with source
29+ presence, policies, generated registration identities, and initial unified
30+ state. Write, fsync, rename, then fsync the directory before attempting state
31+ publication.
32+- Write `state.json` with the same atomic sequence. Hosts, policies, snapshot,
33+ and manifest share one rename commit point. Only then return a ready store.
34+- If state is absent on restart, validate and finish the recovery record without
35+ resampling identities or rereading changed sources. If state exists but is
36+ corrupt, fail closed; never silently roll authority back.
37+- Keep recovery and original inputs indefinitely. Rollback is an explicit
38+ offline operator action using the original bytes, not automatic fallback.
39+ P2.3 does not retire or delete production files.
40+
41+A failed mutation poisons the handle because rename may already have happened.
42+Close and reopen to resolve that outcome; do not blindly retry. Fault tests cover
43+both files at write/fsync/rename/directory-fsync boundaries. They simulate errors
44+and process reopening, not hardware power-loss behavior of a particular disk.
45+
46+The file budget is 16 MiB per JSON file (including the base64 rollback inputs in
47+the recovery file). Inputs are bounded before decoding, must be regular files,
48+and cannot be symlinks. JSON rejects duplicate/unknown fields, trailing content,
49+invalid UTF-8, excessive nesting, and invalid domain/catalogue values.
50+
51+## Authority
52+
53+`BrokerHostStore` separates durable membership from advisory observations.
54+Host replacement uses a store revision CAS. Policy changes require a changed
55+registration identity/generation; generations cannot regress within an
56+incarnation. Full remove/re-add callers must supply a fresh incarnation.
57+Snapshots cannot add hosts, overwrite retired registrations, regress revisions,
58+or switch epochs after the first successful publication on an open handle.
59+`NewRegistry` requires `BrokerHostStore` explicitly: durable membership is not
60+an optional capability, and the registry never runs on a store that cannot
61+answer `LoadHosts`. Startup intersects cached observations with authoritative
62+membership, and creates unknown projections for registrations without
63+observations. An observation is adopted only when it satisfies the same durable
64+projection rules the store enforces (`ports.ValidateDurableHostProjection`):
65+availability and failure kind inside their closed ranges, and a catalogue-valid
66+session inventory for the exact incarnation and success time. Anything else is
67+typed as an invalid response, so a published snapshot is never one the store
68+would reject and silently never persist.
69+
70+Store failures are classified with `internal/ports` sentinels:
71+`ErrBrokerHostConflict` (refresh the revision and retry),
72+`ErrBrokerStoreLocked` (another owner is live), and
73+`ErrBrokerStoreInvalidState` (resolve offline from the recovery record; state is
74+never silently replaced or rolled back). Malformed caller membership passed to
75+`ReplaceHosts` is validated before the CAS and returned as a plain caller error,
76+never as `ErrBrokerStoreInvalidState`; the durable state is unchanged.
77+
78+Unbound cache v2/v3 records remain recoverable but cannot seed registration
79+identity. A bound cache entry seeds only the exact surviving host incarnation;
80+legacy cache schemas do not carry a registration generation. The imported host
81+record remains authoritative for generation. Transient checking/error objects
82+are never persisted. Retirement tombstones are process-local fencing state and
83+are never durable: the registry's persistable copy and the store both clear
84+`Snapshot.Removed` before writing, so reopening sees membership without stale
85+tombstones. No environment, launch, or trust policy is guessed.
86+
87+Production cutover, lifecycle supervision, and IPC composition remain later
88+phases. Close ordering must drain registry snapshot writers before releasing the
89+store; the store does not own those workers.
@@ -0,0 +1,89 @@
1+# Offline broker store (Plan 001 P2.3)
2+
3+`OpenOffline` is deliberately not wired into application composition. Supply a
4+private destination directory, immutable copies of any legacy inputs, and an
5+explicit policy for each imported endpoint. An empty input path means no input;
6+a named missing, empty, corrupt, or unsupported file fails closed. The lifetime
7+lock is `hosts.json.lock`, the exact lock file the legacy host writer takes, so
8+a store opened on a directory that live writer also uses excludes it for the
9+lifetime of the store. The legacy cache source has no such lock — its writer
10+takes none — so a cache source must be immutable or quiesced before migration.
11+Never point migration at a cache file that a writer can still mutate.
12+
13+Each source's recovery manifest entry records its generic role label
14+(`membership` for the host registry, `observations` for the catalogue cache),
15+whether the source was supplied at all, and the SHA-256 of its bytes. Presence
16+is separate from the digest because an absent source and an empty one can share
17+a digest; a record that claims a source was absent while retaining its bytes is
18+rejected.
19+
20+## Transaction and recovery
21+
22+- Hold `hosts.json.lock` with Linux nonblocking exclusive `flock` until `Close`.
23+ The name matches the legacy host writer's lock, so the store and that writer
24+ exclude each other; the legacy catalog cache writer takes no lock at all.
25+- Validate bounded legacy hosts v1/v2/v3 and cache v2/v3/v4 without calling their
26+ existing writer adapters. Cache decoding freezes the existing exact legacy
27+ DTOs here; count-only v2 inventories are deliberately unsupported.
28+- Write `recovery.json`: original bytes, SHA-256 migration manifest with source
29+ presence, policies, generated registration identities, and initial unified
30+ state. Write, fsync, rename, then fsync the directory before attempting state
31+ publication.
32+- Write `state.json` with the same atomic sequence. Hosts, policies, snapshot,
33+ and manifest share one rename commit point. Only then return a ready store.
34+- If state is absent on restart, validate and finish the recovery record without
35+ resampling identities or rereading changed sources. If state exists but is
36+ corrupt, fail closed; never silently roll authority back.
37+- Keep recovery and original inputs indefinitely. Rollback is an explicit
38+ offline operator action using the original bytes, not automatic fallback.
39+ P2.3 does not retire or delete production files.
40+
41+A failed mutation poisons the handle because rename may already have happened.
42+Close and reopen to resolve that outcome; do not blindly retry. Fault tests cover
43+both files at write/fsync/rename/directory-fsync boundaries. They simulate errors
44+and process reopening, not hardware power-loss behavior of a particular disk.
45+
46+The file budget is 16 MiB per JSON file (including the base64 rollback inputs in
47+the recovery file). Inputs are bounded before decoding, must be regular files,
48+and cannot be symlinks. JSON rejects duplicate/unknown fields, trailing content,
49+invalid UTF-8, excessive nesting, and invalid domain/catalogue values.
50+
51+## Authority
52+
53+`BrokerHostStore` separates durable membership from advisory observations.
54+Host replacement uses a store revision CAS. Policy changes require a changed
55+registration identity/generation; generations cannot regress within an
56+incarnation. Full remove/re-add callers must supply a fresh incarnation.
57+Snapshots cannot add hosts, overwrite retired registrations, regress revisions,
58+or switch epochs after the first successful publication on an open handle.
59+`NewRegistry` requires `BrokerHostStore` explicitly: durable membership is not
60+an optional capability, and the registry never runs on a store that cannot
61+answer `LoadHosts`. Startup intersects cached observations with authoritative
62+membership, and creates unknown projections for registrations without
63+observations. An observation is adopted only when it satisfies the same durable
64+projection rules the store enforces (`ports.ValidateDurableHostProjection`):
65+availability and failure kind inside their closed ranges, and a catalogue-valid
66+session inventory for the exact incarnation and success time. Anything else is
67+typed as an invalid response, so a published snapshot is never one the store
68+would reject and silently never persist.
69+
70+Store failures are classified with `internal/ports` sentinels:
71+`ErrBrokerHostConflict` (refresh the revision and retry),
72+`ErrBrokerStoreLocked` (another owner is live), and
73+`ErrBrokerStoreInvalidState` (resolve offline from the recovery record; state is
74+never silently replaced or rolled back). Malformed caller membership passed to
75+`ReplaceHosts` is validated before the CAS and returned as a plain caller error,
76+never as `ErrBrokerStoreInvalidState`; the durable state is unchanged.
77+
78+Unbound cache v2/v3 records remain recoverable but cannot seed registration
79+identity. A bound cache entry seeds only the exact surviving host incarnation;
80+legacy cache schemas do not carry a registration generation. The imported host
81+record remains authoritative for generation. Transient checking/error objects
82+are never persisted. Retirement tombstones are process-local fencing state and
83+are never durable: the registry's persistable copy and the store both clear
84+`Snapshot.Removed` before writing, so reopening sees membership without stale
85+tombstones. No environment, launch, or trust policy is guessed.
86+
87+Production cutover, lifecycle supervision, and IPC composition remain later
88+phases. Close ordering must drain registry snapshot writers before releasing the
89+store; the store does not own those workers.
A internal/adapters/brokerstore/legacy.go +268 -0
@@ -0,0 +1,268 @@
1+package brokerstore
2+
3+import (
4+ "bytes"
5+ "crypto/rand"
6+ "encoding/hex"
7+ "encoding/json"
8+ "errors"
9+ "fmt"
10+ "io"
11+ "os"
12+ "syscall"
13+ "unicode/utf8"
14+
15+ "github.com/bnema/vev/internal/domain"
16+ "github.com/bnema/vev/internal/ports"
17+)
18+
19+func readBounded(path string) ([]byte, error) {
20+ fd, err := syscall.Open(path, syscall.O_RDONLY|syscall.O_CLOEXEC|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0)
21+ if err != nil {
22+ return nil, err
23+ }
24+ f := os.NewFile(uintptr(fd), path)
25+ defer f.Close()
26+ st, err := f.Stat()
27+ if err != nil {
28+ return nil, err
29+ }
30+ if !st.Mode().IsRegular() || st.Size() > MaxFileBytes {
31+ return nil, errors.New("brokerstore: invalid file type or size")
32+ }
33+ b, err := io.ReadAll(io.LimitReader(f, MaxFileBytes+1))
34+ if len(b) > MaxFileBytes {
35+ return nil, errors.New("brokerstore: oversized input")
36+ }
37+ return b, err
38+}
39+
40+// strict rejects duplicate keys as well as unknown fields, trailing JSON,
41+// invalid UTF-8 and excessive nesting. Byte bounds apply before decoding.
42+func strict(raw []byte, dst any) error {
43+ if len(raw) > MaxFileBytes || !utf8.Valid(raw) {
44+ return errors.New("brokerstore: invalid JSON size or encoding")
45+ }
46+ scan := json.NewDecoder(bytes.NewReader(raw))
47+ if err := scanValue(scan, 0); err != nil {
48+ return err
49+ }
50+ if _, err := scan.Token(); err != io.EOF {
51+ return errors.New("brokerstore: trailing JSON")
52+ }
53+ dec := json.NewDecoder(bytes.NewReader(raw))
54+ dec.DisallowUnknownFields()
55+ return dec.Decode(dst)
56+}
57+func scanValue(d *json.Decoder, depth int) error {
58+ if depth > 64 {
59+ return errors.New("brokerstore: JSON nesting limit")
60+ }
61+ t, err := d.Token()
62+ if err != nil {
63+ return err
64+ }
65+ delim, ok := t.(json.Delim)
66+ if !ok {
67+ return nil
68+ }
69+ switch delim {
70+ case '{':
71+ seen := map[string]bool{}
72+ for d.More() {
73+ key, err := d.Token()
74+ if err != nil {
75+ return err
76+ }
77+ k, ok := key.(string)
78+ if !ok || seen[k] {
79+ return errors.New("brokerstore: duplicate JSON key")
80+ }
81+ seen[k] = true
82+ if err := scanValue(d, depth+1); err != nil {
83+ return err
84+ }
85+ }
86+ case '[':
87+ for d.More() {
88+ if err := scanValue(d, depth+1); err != nil {
89+ return err
90+ }
91+ }
92+ default:
93+ return errors.New("brokerstore: unexpected delimiter")
94+ }
95+ _, err = d.Token()
96+ return err
97+}
98+func source(path string) ([]byte, error) {
99+ if path == "" {
100+ return nil, nil
101+ }
102+ return readBounded(path)
103+}
104+func migrate(o Options) (recovery, error) {
105+ var r recovery
106+ var err error
107+ if r.Hosts, err = source(o.LegacyHosts); err != nil {
108+ return r, err
109+ }
110+ if r.Cache, err = source(o.LegacyCache); err != nil {
111+ return r, err
112+ }
113+ r.State.Manifest = manifest{Version: 1, Sources: []manifestSource{
114+ {Label: manifestMembership, Present: o.LegacyHosts != "", SHA256: digest(r.Hosts)},
115+ {Label: manifestObservations, Present: o.LegacyCache != "", SHA256: digest(r.Cache)},
116+ }}
117+ r.State.Hosts = ports.BrokerHosts{Revision: 1}
118+ if o.LegacyHosts != "" {
119+ r.State.Hosts.Hosts, err = decodeHosts(r.Hosts, o.Policies)
120+ if err != nil {
121+ return r, err
122+ }
123+ }
124+ if o.LegacyCache != "" {
125+ entries, err := decodeCache(r.Cache)
126+ if err != nil {
127+ return r, err
128+ }
129+ for _, e := range entries {
130+ for _, h := range r.State.Hosts.Hosts {
131+ // Unbound v2/v3 cache data is validated and backed up, never promoted
132+ // to authority. v4 can seed only its exact surviving incarnation.
133+ if e.Host == h.Registration.Endpoint && e.Incarnation == h.Registration.Incarnation {
134+ r.State.Snapshot.Hosts = append(r.State.Snapshot.Hosts, ports.RemoteHostSnapshot{Endpoint: e.Host, Registration: h.Registration, Availability: domain.RemoteAvailabilityUnknown, InventoryKnown: true, LastSuccess: e.FetchedAt, Sessions: e.Sessions})
135+ }
136+ }
137+ }
138+ }
139+ if len(r.State.Snapshot.Hosts) > 0 {
140+ var epoch [8]byte
141+ if _, err = rand.Read(epoch[:]); err != nil {
142+ return r, err
143+ }
144+ var n uint64
145+ for _, b := range epoch {
146+ n = n<<8 | uint64(b)
147+ }
148+ if n == 0 {
149+ n = 1
150+ }
151+ r.State.Snapshot.Epoch = ports.BrokerEpoch(n)
152+ r.State.Snapshot.Revision = 1
153+ }
154+ return r, validate(r.State)
155+}
156+
157+type legacyRecord struct {
158+ Endpoint string `json:"endpoint"`
159+ Incarnation string `json:"incarnation"`
160+ Generation uint64 `json:"generation"`
161+}
162+
163+func decodeHosts(raw []byte, policies map[string]ports.BrokerPolicy) ([]ports.BrokerHostRecord, error) {
164+ var header struct {
165+ Version int `json:"version"`
166+ }
167+ if err := json.Unmarshal(raw, &header); err != nil {
168+ return nil, err
169+ }
170+ var pinned, learned []legacyRecord
171+ switch header.Version {
172+ case 1:
173+ var f struct {
174+ Version int `json:"version"`
175+ Hosts []string `json:"hosts"`
176+ }
177+ if err := strict(raw, &f); err != nil {
178+ return nil, err
179+ }
180+ if f.Hosts == nil {
181+ return nil, errors.New("brokerstore: missing hosts")
182+ }
183+ for _, e := range f.Hosts {
184+ learned = append(learned, legacyRecord{Endpoint: e})
185+ }
186+ case 2:
187+ var f struct {
188+ Version int `json:"version"`
189+ Pinned []string `json:"pinned"`
190+ Learned []string `json:"learned"`
191+ }
192+ if err := strict(raw, &f); err != nil {
193+ return nil, err
194+ }
195+ if f.Pinned == nil || f.Learned == nil {
196+ return nil, errors.New("brokerstore: missing memberships")
197+ }
198+ for _, e := range f.Pinned {
199+ pinned = append(pinned, legacyRecord{Endpoint: e})
200+ }
201+ for _, e := range f.Learned {
202+ learned = append(learned, legacyRecord{Endpoint: e})
203+ }
204+ case 3:
205+ var f struct {
206+ Version int `json:"version"`
207+ Pinned []legacyRecord `json:"pinned"`
208+ Learned []legacyRecord `json:"learned"`
209+ }
210+ if err := strict(raw, &f); err != nil {
211+ return nil, err
212+ }
213+ if f.Pinned == nil || f.Learned == nil {
214+ return nil, errors.New("brokerstore: missing memberships")
215+ }
216+ pinned, learned = f.Pinned, f.Learned
217+ default:
218+ return nil, fmt.Errorf("brokerstore: unsupported hosts version %d", header.Version)
219+ }
220+ if len(pinned) > ports.BrokerMaxHosts || len(learned) > ports.BrokerMaxHosts {
221+ return nil, errors.New("brokerstore: too many hosts")
222+ }
223+ var out []ports.BrokerHostRecord
224+ indices := map[string]int{}
225+ for group, records := range [][]legacyRecord{pinned, learned} {
226+ seen := map[string]bool{}
227+ for _, record := range records {
228+ if seen[record.Endpoint] {
229+ return nil, errors.New("brokerstore: duplicate membership")
230+ }
231+ seen[record.Endpoint] = true
232+ reg := domain.RemoteRegistration{Endpoint: record.Endpoint, Generation: domain.RemoteGeneration(record.Generation)}
233+ if header.Version == 3 {
234+ b, err := hex.DecodeString(record.Incarnation)
235+ if err != nil || len(b) != 16 {
236+ return nil, errors.New("brokerstore: invalid incarnation")
237+ }
238+ copy(reg.Incarnation[:], b)
239+ }
240+ if index, ok := indices[record.Endpoint]; ok {
241+ if header.Version == 3 && !reg.Equal(out[index].Registration) {
242+ return nil, errors.New("brokerstore: conflicting registration")
243+ }
244+ out[index].Learned = true
245+ continue
246+ }
247+ if header.Version != 3 {
248+ if _, err := rand.Read(reg.Incarnation[:]); err != nil {
249+ return nil, err
250+ }
251+ reg.Generation = 1
252+ }
253+ if err := reg.Validate(); err != nil {
254+ return nil, err
255+ }
256+ policy, ok := policies[record.Endpoint]
257+ if !ok {
258+ return nil, fmt.Errorf("brokerstore: explicit policy required for %q", record.Endpoint)
259+ }
260+ if err := policy.Validate(); err != nil {
261+ return nil, err
262+ }
263+ indices[record.Endpoint] = len(out)
264+ out = append(out, ports.BrokerHostRecord{Registration: reg, Pinned: group == 0, Learned: group == 1, Policy: policy})
265+ }
266+ }
267+ return out, nil
268+}
@@ -0,0 +1,268 @@
1+package brokerstore
2+
3+import (
4+ "bytes"
5+ "crypto/rand"
6+ "encoding/hex"
7+ "encoding/json"
8+ "errors"
9+ "fmt"
10+ "io"
11+ "os"
12+ "syscall"
13+ "unicode/utf8"
14+
15+ "github.com/bnema/vev/internal/domain"
16+ "github.com/bnema/vev/internal/ports"
17+)
18+
19+func readBounded(path string) ([]byte, error) {
20+ fd, err := syscall.Open(path, syscall.O_RDONLY|syscall.O_CLOEXEC|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0)
21+ if err != nil {
22+ return nil, err
23+ }
24+ f := os.NewFile(uintptr(fd), path)
25+ defer f.Close()
26+ st, err := f.Stat()
27+ if err != nil {
28+ return nil, err
29+ }
30+ if !st.Mode().IsRegular() || st.Size() > MaxFileBytes {
31+ return nil, errors.New("brokerstore: invalid file type or size")
32+ }
33+ b, err := io.ReadAll(io.LimitReader(f, MaxFileBytes+1))
34+ if len(b) > MaxFileBytes {
35+ return nil, errors.New("brokerstore: oversized input")
36+ }
37+ return b, err
38+}
39+
40+// strict rejects duplicate keys as well as unknown fields, trailing JSON,
41+// invalid UTF-8 and excessive nesting. Byte bounds apply before decoding.
42+func strict(raw []byte, dst any) error {
43+ if len(raw) > MaxFileBytes || !utf8.Valid(raw) {
44+ return errors.New("brokerstore: invalid JSON size or encoding")
45+ }
46+ scan := json.NewDecoder(bytes.NewReader(raw))
47+ if err := scanValue(scan, 0); err != nil {
48+ return err
49+ }
50+ if _, err := scan.Token(); err != io.EOF {
51+ return errors.New("brokerstore: trailing JSON")
52+ }
53+ dec := json.NewDecoder(bytes.NewReader(raw))
54+ dec.DisallowUnknownFields()
55+ return dec.Decode(dst)
56+}
57+func scanValue(d *json.Decoder, depth int) error {
58+ if depth > 64 {
59+ return errors.New("brokerstore: JSON nesting limit")
60+ }
61+ t, err := d.Token()
62+ if err != nil {
63+ return err
64+ }
65+ delim, ok := t.(json.Delim)
66+ if !ok {
67+ return nil
68+ }
69+ switch delim {
70+ case '{':
71+ seen := map[string]bool{}
72+ for d.More() {
73+ key, err := d.Token()
74+ if err != nil {
75+ return err
76+ }
77+ k, ok := key.(string)
78+ if !ok || seen[k] {
79+ return errors.New("brokerstore: duplicate JSON key")
80+ }
81+ seen[k] = true
82+ if err := scanValue(d, depth+1); err != nil {
83+ return err
84+ }
85+ }
86+ case '[':
87+ for d.More() {
88+ if err := scanValue(d, depth+1); err != nil {
89+ return err
90+ }
91+ }
92+ default:
93+ return errors.New("brokerstore: unexpected delimiter")
94+ }
95+ _, err = d.Token()
96+ return err
97+}
98+func source(path string) ([]byte, error) {
99+ if path == "" {
100+ return nil, nil
101+ }
102+ return readBounded(path)
103+}
104+func migrate(o Options) (recovery, error) {
105+ var r recovery
106+ var err error
107+ if r.Hosts, err = source(o.LegacyHosts); err != nil {
108+ return r, err
109+ }
110+ if r.Cache, err = source(o.LegacyCache); err != nil {
111+ return r, err
112+ }
113+ r.State.Manifest = manifest{Version: 1, Sources: []manifestSource{
114+ {Label: manifestMembership, Present: o.LegacyHosts != "", SHA256: digest(r.Hosts)},
115+ {Label: manifestObservations, Present: o.LegacyCache != "", SHA256: digest(r.Cache)},
116+ }}
117+ r.State.Hosts = ports.BrokerHosts{Revision: 1}
118+ if o.LegacyHosts != "" {
119+ r.State.Hosts.Hosts, err = decodeHosts(r.Hosts, o.Policies)
120+ if err != nil {
121+ return r, err
122+ }
123+ }
124+ if o.LegacyCache != "" {
125+ entries, err := decodeCache(r.Cache)
126+ if err != nil {
127+ return r, err
128+ }
129+ for _, e := range entries {
130+ for _, h := range r.State.Hosts.Hosts {
131+ // Unbound v2/v3 cache data is validated and backed up, never promoted
132+ // to authority. v4 can seed only its exact surviving incarnation.
133+ if e.Host == h.Registration.Endpoint && e.Incarnation == h.Registration.Incarnation {
134+ r.State.Snapshot.Hosts = append(r.State.Snapshot.Hosts, ports.RemoteHostSnapshot{Endpoint: e.Host, Registration: h.Registration, Availability: domain.RemoteAvailabilityUnknown, InventoryKnown: true, LastSuccess: e.FetchedAt, Sessions: e.Sessions})
135+ }
136+ }
137+ }
138+ }
139+ if len(r.State.Snapshot.Hosts) > 0 {
140+ var epoch [8]byte
141+ if _, err = rand.Read(epoch[:]); err != nil {
142+ return r, err
143+ }
144+ var n uint64
145+ for _, b := range epoch {
146+ n = n<<8 | uint64(b)
147+ }
148+ if n == 0 {
149+ n = 1
150+ }
151+ r.State.Snapshot.Epoch = ports.BrokerEpoch(n)
152+ r.State.Snapshot.Revision = 1
153+ }
154+ return r, validate(r.State)
155+}
156+
157+type legacyRecord struct {
158+ Endpoint string `json:"endpoint"`
159+ Incarnation string `json:"incarnation"`
160+ Generation uint64 `json:"generation"`
161+}
162+
163+func decodeHosts(raw []byte, policies map[string]ports.BrokerPolicy) ([]ports.BrokerHostRecord, error) {
164+ var header struct {
165+ Version int `json:"version"`
166+ }
167+ if err := json.Unmarshal(raw, &header); err != nil {
168+ return nil, err
169+ }
170+ var pinned, learned []legacyRecord
171+ switch header.Version {
172+ case 1:
173+ var f struct {
174+ Version int `json:"version"`
175+ Hosts []string `json:"hosts"`
176+ }
177+ if err := strict(raw, &f); err != nil {
178+ return nil, err
179+ }
180+ if f.Hosts == nil {
181+ return nil, errors.New("brokerstore: missing hosts")
182+ }
183+ for _, e := range f.Hosts {
184+ learned = append(learned, legacyRecord{Endpoint: e})
185+ }
186+ case 2:
187+ var f struct {
188+ Version int `json:"version"`
189+ Pinned []string `json:"pinned"`
190+ Learned []string `json:"learned"`
191+ }
192+ if err := strict(raw, &f); err != nil {
193+ return nil, err
194+ }
195+ if f.Pinned == nil || f.Learned == nil {
196+ return nil, errors.New("brokerstore: missing memberships")
197+ }
198+ for _, e := range f.Pinned {
199+ pinned = append(pinned, legacyRecord{Endpoint: e})
200+ }
201+ for _, e := range f.Learned {
202+ learned = append(learned, legacyRecord{Endpoint: e})
203+ }
204+ case 3:
205+ var f struct {
206+ Version int `json:"version"`
207+ Pinned []legacyRecord `json:"pinned"`
208+ Learned []legacyRecord `json:"learned"`
209+ }
210+ if err := strict(raw, &f); err != nil {
211+ return nil, err
212+ }
213+ if f.Pinned == nil || f.Learned == nil {
214+ return nil, errors.New("brokerstore: missing memberships")
215+ }
216+ pinned, learned = f.Pinned, f.Learned
217+ default:
218+ return nil, fmt.Errorf("brokerstore: unsupported hosts version %d", header.Version)
219+ }
220+ if len(pinned) > ports.BrokerMaxHosts || len(learned) > ports.BrokerMaxHosts {
221+ return nil, errors.New("brokerstore: too many hosts")
222+ }
223+ var out []ports.BrokerHostRecord
224+ indices := map[string]int{}
225+ for group, records := range [][]legacyRecord{pinned, learned} {
226+ seen := map[string]bool{}
227+ for _, record := range records {
228+ if seen[record.Endpoint] {
229+ return nil, errors.New("brokerstore: duplicate membership")
230+ }
231+ seen[record.Endpoint] = true
232+ reg := domain.RemoteRegistration{Endpoint: record.Endpoint, Generation: domain.RemoteGeneration(record.Generation)}
233+ if header.Version == 3 {
234+ b, err := hex.DecodeString(record.Incarnation)
235+ if err != nil || len(b) != 16 {
236+ return nil, errors.New("brokerstore: invalid incarnation")
237+ }
238+ copy(reg.Incarnation[:], b)
239+ }
240+ if index, ok := indices[record.Endpoint]; ok {
241+ if header.Version == 3 && !reg.Equal(out[index].Registration) {
242+ return nil, errors.New("brokerstore: conflicting registration")
243+ }
244+ out[index].Learned = true
245+ continue
246+ }
247+ if header.Version != 3 {
248+ if _, err := rand.Read(reg.Incarnation[:]); err != nil {
249+ return nil, err
250+ }
251+ reg.Generation = 1
252+ }
253+ if err := reg.Validate(); err != nil {
254+ return nil, err
255+ }
256+ policy, ok := policies[record.Endpoint]
257+ if !ok {
258+ return nil, fmt.Errorf("brokerstore: explicit policy required for %q", record.Endpoint)
259+ }
260+ if err := policy.Validate(); err != nil {
261+ return nil, err
262+ }
263+ indices[record.Endpoint] = len(out)
264+ out = append(out, ports.BrokerHostRecord{Registration: reg, Pinned: group == 0, Learned: group == 1, Policy: policy})
265+ }
266+ }
267+ return out, nil
268+}
A internal/adapters/brokerstore/legacy_cache.go +258 -0
@@ -0,0 +1,258 @@
1+package brokerstore
2+
3+import (
4+ "encoding/hex"
5+ "encoding/json"
6+ "fmt"
7+ "sort"
8+ "time"
9+ "unicode/utf8"
10+
11+ "github.com/bnema/vev/internal/domain"
12+ "github.com/bnema/vev/internal/protocol/catalogue"
13+)
14+
15+const (
16+ catalogCacheFileVersion = 4
17+ catalogCacheLegacyFileVersion = 3
18+ catalogCacheExactFileVersion = 2
19+)
20+
21+type catalogCacheFile struct {
22+ Version *int `json:"version"`
23+ Hosts *[]catalogCacheHost `json:"hosts"`
24+}
25+
26+type catalogCacheVersion struct {
27+ Version *int `json:"version"`
28+}
29+
30+type catalogCacheHost struct {
31+ Target *string `json:"target"`
32+ FetchedAtUnixNano *int64 `json:"fetched_at_unix_nano"`
33+ Incarnation *string `json:"incarnation,omitempty"`
34+ Sessions *[]catalogCacheSession `json:"sessions"`
35+}
36+
37+type catalogCacheSession struct {
38+ LifecycleID *domain.SessionLifecycleID `json:"lifecycle_id"`
39+ Name *string `json:"name"`
40+ State *catalogue.RemoteCatalogSessionState `json:"state"`
41+ Ephemeral *bool `json:"ephemeral"`
42+ LastUsedSeq *uint64 `json:"last_used_seq,omitempty"`
43+ Tabs *[]catalogue.RemoteCatalogTab `json:"tabs"`
44+ ActiveTabID *string `json:"active_tab_id,omitempty"`
45+ Attached *bool `json:"attached"`
46+}
47+
48+type catalogCacheFileV2 struct {
49+ Version *int `json:"version"`
50+ Hosts *[]catalogCacheHostV2 `json:"hosts"`
51+}
52+
53+type catalogCacheHostV2 struct {
54+ Target *string `json:"target"`
55+ FetchedAtUnixNano *int64 `json:"fetched_at_unix_nano"`
56+ Sessions *[]catalogCacheSessionV2 `json:"sessions"`
57+}
58+
59+type catalogCacheSessionV2 struct {
60+ LifecycleID *domain.SessionLifecycleID `json:"lifecycle_id,omitempty"`
61+ Name *string `json:"name"`
62+ State *catalogue.RemoteCatalogSessionState `json:"state"`
63+ Ephemeral *bool `json:"ephemeral"`
64+ LastUsedSeq *uint64 `json:"last_used_seq,omitempty"`
65+ Tabs *uint16 `json:"tabs,omitempty"`
66+ TabList *[]catalogue.RemoteCatalogTab `json:"tab_list,omitempty"`
67+ ActiveTabID *string `json:"active_tab_id,omitempty"`
68+ Attached *bool `json:"attached"`
69+}
70+
71+func decodeCache(raw []byte) ([]catalogue.RemoteCatalogCacheEntry, error) {
72+ var version catalogCacheVersion
73+ if err := json.Unmarshal(raw, &version); err != nil {
74+ return nil, err
75+ }
76+ if version.Version == nil {
77+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: missing version")
78+ }
79+
80+ var file catalogCacheFile
81+ requireIncarnation := false
82+ switch *version.Version {
83+ case catalogCacheFileVersion:
84+ requireIncarnation = true
85+ if err := strict(raw, &file); err != nil {
86+ return nil, err
87+ }
88+ case catalogCacheLegacyFileVersion:
89+ if err := strict(raw, &file); err != nil {
90+ return nil, err
91+ }
92+ case catalogCacheExactFileVersion:
93+ var legacy catalogCacheFileV2
94+ if err := strict(raw, &legacy); err != nil {
95+ return nil, err
96+ }
97+ migrated, err := migrateExactCatalogCacheV2(legacy)
98+ if err != nil {
99+ return nil, err
100+ }
101+ file = migrated
102+ default:
103+ return nil, fmt.Errorf("remote catalog cache: unsupported cache file version %d", *version.Version)
104+ }
105+ if file.Hosts == nil {
106+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: missing hosts")
107+ }
108+
109+ entries := make([]catalogue.RemoteCatalogCacheEntry, 0, len(*file.Hosts))
110+ for _, host := range *file.Hosts {
111+ if host.Target == nil || host.FetchedAtUnixNano == nil || host.Sessions == nil {
112+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: missing host fields")
113+ }
114+ if !utf8.ValidString(*host.Target) {
115+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: invalid UTF-8")
116+ }
117+ if *host.FetchedAtUnixNano <= 0 {
118+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: non-positive fetched time")
119+ }
120+ entry := catalogue.RemoteCatalogCacheEntry{
121+ Host: *host.Target,
122+ FetchedAt: time.Unix(0, *host.FetchedAtUnixNano),
123+ Sessions: make([]catalogue.RemoteCatalogSession, 0, len(*host.Sessions)),
124+ }
125+ if host.Incarnation != nil {
126+ raw, err := hex.DecodeString(*host.Incarnation)
127+ if err != nil || len(raw) != 16 {
128+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: invalid incarnation for %q", *host.Target)
129+ }
130+ copy(entry.Incarnation[:], raw)
131+ } else if requireIncarnation {
132+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: missing incarnation for %q", *host.Target)
133+ }
134+ // A zero incarnation is unbound advisory data: it loads but must
135+ // never seed a registration.
136+ for _, session := range *host.Sessions {
137+ if session.LifecycleID == nil || session.Name == nil || session.State == nil || session.Ephemeral == nil || session.Attached == nil || session.Tabs == nil {
138+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: missing session fields")
139+ }
140+ if !utf8.ValidString(*session.Name) || !utf8.ValidString(string(*session.State)) {
141+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: invalid UTF-8")
142+ }
143+ tabs := make([]catalogue.RemoteCatalogTab, len(*session.Tabs))
144+ copy(tabs, *session.Tabs)
145+ decoded := catalogue.RemoteCatalogSession{
146+ LifecycleID: *session.LifecycleID,
147+ Name: *session.Name,
148+ State: *session.State,
149+ Ephemeral: *session.Ephemeral,
150+ Tabs: tabs,
151+ Attached: *session.Attached,
152+ }
153+ if session.LastUsedSeq != nil {
154+ decoded.LastUsedSeq = *session.LastUsedSeq
155+ }
156+ if session.ActiveTabID != nil {
157+ decoded.ActiveTabID = *session.ActiveTabID
158+ }
159+ entry.Sessions = append(entry.Sessions, decoded)
160+ }
161+ entries = append(entries, entry)
162+ }
163+ normalized, err := normalizeCatalogCacheEntries(entries)
164+ if err != nil {
165+ return nil, err
166+ }
167+ return normalized, nil
168+}
169+
170+func migrateExactCatalogCacheV2(legacy catalogCacheFileV2) (catalogCacheFile, error) {
171+ if legacy.Version == nil || *legacy.Version != catalogCacheExactFileVersion || legacy.Hosts == nil {
172+ return catalogCacheFile{}, fmt.Errorf("remote catalog cache: malformed version 2 cache file")
173+ }
174+ hosts := make([]catalogCacheHost, 0, len(*legacy.Hosts))
175+ for _, host := range *legacy.Hosts {
176+ if host.Sessions == nil {
177+ return catalogCacheFile{}, fmt.Errorf("remote catalog cache: malformed cache file: missing host fields")
178+ }
179+ sessions := make([]catalogCacheSession, 0, len(*host.Sessions))
180+ for _, session := range *host.Sessions {
181+ if session.Tabs != nil || session.TabList == nil {
182+ return catalogCacheFile{}, fmt.Errorf("remote catalog cache: unsupported ambiguous version 2 tab count")
183+ }
184+ sessions = append(sessions, catalogCacheSession{
185+ LifecycleID: session.LifecycleID,
186+ Name: session.Name,
187+ State: session.State,
188+ Ephemeral: session.Ephemeral,
189+ LastUsedSeq: session.LastUsedSeq,
190+ Tabs: session.TabList,
191+ ActiveTabID: session.ActiveTabID,
192+ Attached: session.Attached,
193+ })
194+ }
195+ hosts = append(hosts, catalogCacheHost{
196+ Target: host.Target, FetchedAtUnixNano: host.FetchedAtUnixNano, Sessions: &sessions,
197+ })
198+ }
199+ version := catalogCacheFileVersion
200+ return catalogCacheFile{Version: &version, Hosts: &hosts}, nil
201+}
202+
203+func normalizeCatalogCacheEntries(entries []catalogue.RemoteCatalogCacheEntry) ([]catalogue.RemoteCatalogCacheEntry, error) {
204+ if err := catalogue.ValidateRemoteCatalogCacheEntries(entries); err != nil {
205+ return nil, err
206+ }
207+ normalized := make([]catalogue.RemoteCatalogCacheEntry, 0, len(entries))
208+ hosts := make(map[string]struct{}, len(entries))
209+ for _, entry := range entries {
210+ if err := domain.ValidateRemoteHostTarget(entry.Host); err != nil {
211+ return nil, fmt.Errorf("remote catalog cache: invalid host %q: %w", entry.Host, err)
212+ }
213+ if entry.FetchedAt.UnixNano() <= 0 {
214+ return nil, fmt.Errorf("remote catalog cache: host %q has non-positive fetched time", entry.Host)
215+ }
216+ if _, duplicate := hosts[entry.Host]; duplicate {
217+ return nil, fmt.Errorf("remote catalog cache: duplicate host %q", entry.Host)
218+ }
219+ hosts[entry.Host] = struct{}{}
220+
221+ copyEntry := catalogue.RemoteCatalogCacheEntry{
222+ Host: entry.Host,
223+ FetchedAt: entry.FetchedAt,
224+ Incarnation: entry.Incarnation,
225+ Sessions: make([]catalogue.RemoteCatalogSession, 0, len(entry.Sessions)),
226+ }
227+ sessions := make(map[string]struct{}, len(entry.Sessions))
228+ for _, session := range entry.Sessions {
229+ if !utf8.ValidString(session.Name) || !utf8.ValidString(string(session.State)) {
230+ return nil, fmt.Errorf("remote catalog cache: session is not valid UTF-8")
231+ }
232+ if err := domain.ValidateSessionName(session.Name); err != nil {
233+ return nil, fmt.Errorf("remote catalog cache: invalid session %q: %w", session.Name, err)
234+ }
235+ if !session.State.Valid() {
236+ return nil, fmt.Errorf("remote catalog cache: invalid session state %q", session.State)
237+ }
238+ if _, duplicate := sessions[session.Name]; duplicate {
239+ return nil, fmt.Errorf("remote catalog cache: duplicate session %q for host %q", session.Name, entry.Host)
240+ }
241+ sessions[session.Name] = struct{}{}
242+ copySession := session
243+ copySession.Tabs = make([]catalogue.RemoteCatalogTab, len(session.Tabs))
244+ for i, tab := range session.Tabs {
245+ copySession.Tabs[i] = catalogue.RemoteCatalogTab{ID: tab.ID, Index: tab.Index, Name: tab.Name}
246+ }
247+ copyEntry.Sessions = append(copyEntry.Sessions, copySession)
248+ }
249+ sort.Slice(copyEntry.Sessions, func(i, j int) bool {
250+ return copyEntry.Sessions[i].Name < copyEntry.Sessions[j].Name
251+ })
252+ normalized = append(normalized, copyEntry)
253+ }
254+ sort.Slice(normalized, func(i, j int) bool {
255+ return normalized[i].Host < normalized[j].Host
256+ })
257+ return normalized, nil
258+}
@@ -0,0 +1,258 @@
1+package brokerstore
2+
3+import (
4+ "encoding/hex"
5+ "encoding/json"
6+ "fmt"
7+ "sort"
8+ "time"
9+ "unicode/utf8"
10+
11+ "github.com/bnema/vev/internal/domain"
12+ "github.com/bnema/vev/internal/protocol/catalogue"
13+)
14+
15+const (
16+ catalogCacheFileVersion = 4
17+ catalogCacheLegacyFileVersion = 3
18+ catalogCacheExactFileVersion = 2
19+)
20+
21+type catalogCacheFile struct {
22+ Version *int `json:"version"`
23+ Hosts *[]catalogCacheHost `json:"hosts"`
24+}
25+
26+type catalogCacheVersion struct {
27+ Version *int `json:"version"`
28+}
29+
30+type catalogCacheHost struct {
31+ Target *string `json:"target"`
32+ FetchedAtUnixNano *int64 `json:"fetched_at_unix_nano"`
33+ Incarnation *string `json:"incarnation,omitempty"`
34+ Sessions *[]catalogCacheSession `json:"sessions"`
35+}
36+
37+type catalogCacheSession struct {
38+ LifecycleID *domain.SessionLifecycleID `json:"lifecycle_id"`
39+ Name *string `json:"name"`
40+ State *catalogue.RemoteCatalogSessionState `json:"state"`
41+ Ephemeral *bool `json:"ephemeral"`
42+ LastUsedSeq *uint64 `json:"last_used_seq,omitempty"`
43+ Tabs *[]catalogue.RemoteCatalogTab `json:"tabs"`
44+ ActiveTabID *string `json:"active_tab_id,omitempty"`
45+ Attached *bool `json:"attached"`
46+}
47+
48+type catalogCacheFileV2 struct {
49+ Version *int `json:"version"`
50+ Hosts *[]catalogCacheHostV2 `json:"hosts"`
51+}
52+
53+type catalogCacheHostV2 struct {
54+ Target *string `json:"target"`
55+ FetchedAtUnixNano *int64 `json:"fetched_at_unix_nano"`
56+ Sessions *[]catalogCacheSessionV2 `json:"sessions"`
57+}
58+
59+type catalogCacheSessionV2 struct {
60+ LifecycleID *domain.SessionLifecycleID `json:"lifecycle_id,omitempty"`
61+ Name *string `json:"name"`
62+ State *catalogue.RemoteCatalogSessionState `json:"state"`
63+ Ephemeral *bool `json:"ephemeral"`
64+ LastUsedSeq *uint64 `json:"last_used_seq,omitempty"`
65+ Tabs *uint16 `json:"tabs,omitempty"`
66+ TabList *[]catalogue.RemoteCatalogTab `json:"tab_list,omitempty"`
67+ ActiveTabID *string `json:"active_tab_id,omitempty"`
68+ Attached *bool `json:"attached"`
69+}
70+
71+func decodeCache(raw []byte) ([]catalogue.RemoteCatalogCacheEntry, error) {
72+ var version catalogCacheVersion
73+ if err := json.Unmarshal(raw, &version); err != nil {
74+ return nil, err
75+ }
76+ if version.Version == nil {
77+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: missing version")
78+ }
79+
80+ var file catalogCacheFile
81+ requireIncarnation := false
82+ switch *version.Version {
83+ case catalogCacheFileVersion:
84+ requireIncarnation = true
85+ if err := strict(raw, &file); err != nil {
86+ return nil, err
87+ }
88+ case catalogCacheLegacyFileVersion:
89+ if err := strict(raw, &file); err != nil {
90+ return nil, err
91+ }
92+ case catalogCacheExactFileVersion:
93+ var legacy catalogCacheFileV2
94+ if err := strict(raw, &legacy); err != nil {
95+ return nil, err
96+ }
97+ migrated, err := migrateExactCatalogCacheV2(legacy)
98+ if err != nil {
99+ return nil, err
100+ }
101+ file = migrated
102+ default:
103+ return nil, fmt.Errorf("remote catalog cache: unsupported cache file version %d", *version.Version)
104+ }
105+ if file.Hosts == nil {
106+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: missing hosts")
107+ }
108+
109+ entries := make([]catalogue.RemoteCatalogCacheEntry, 0, len(*file.Hosts))
110+ for _, host := range *file.Hosts {
111+ if host.Target == nil || host.FetchedAtUnixNano == nil || host.Sessions == nil {
112+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: missing host fields")
113+ }
114+ if !utf8.ValidString(*host.Target) {
115+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: invalid UTF-8")
116+ }
117+ if *host.FetchedAtUnixNano <= 0 {
118+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: non-positive fetched time")
119+ }
120+ entry := catalogue.RemoteCatalogCacheEntry{
121+ Host: *host.Target,
122+ FetchedAt: time.Unix(0, *host.FetchedAtUnixNano),
123+ Sessions: make([]catalogue.RemoteCatalogSession, 0, len(*host.Sessions)),
124+ }
125+ if host.Incarnation != nil {
126+ raw, err := hex.DecodeString(*host.Incarnation)
127+ if err != nil || len(raw) != 16 {
128+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: invalid incarnation for %q", *host.Target)
129+ }
130+ copy(entry.Incarnation[:], raw)
131+ } else if requireIncarnation {
132+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: missing incarnation for %q", *host.Target)
133+ }
134+ // A zero incarnation is unbound advisory data: it loads but must
135+ // never seed a registration.
136+ for _, session := range *host.Sessions {
137+ if session.LifecycleID == nil || session.Name == nil || session.State == nil || session.Ephemeral == nil || session.Attached == nil || session.Tabs == nil {
138+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: missing session fields")
139+ }
140+ if !utf8.ValidString(*session.Name) || !utf8.ValidString(string(*session.State)) {
141+ return nil, fmt.Errorf("remote catalog cache: malformed cache file: invalid UTF-8")
142+ }
143+ tabs := make([]catalogue.RemoteCatalogTab, len(*session.Tabs))
144+ copy(tabs, *session.Tabs)
145+ decoded := catalogue.RemoteCatalogSession{
146+ LifecycleID: *session.LifecycleID,
147+ Name: *session.Name,
148+ State: *session.State,
149+ Ephemeral: *session.Ephemeral,
150+ Tabs: tabs,
151+ Attached: *session.Attached,
152+ }
153+ if session.LastUsedSeq != nil {
154+ decoded.LastUsedSeq = *session.LastUsedSeq
155+ }
156+ if session.ActiveTabID != nil {
157+ decoded.ActiveTabID = *session.ActiveTabID
158+ }
159+ entry.Sessions = append(entry.Sessions, decoded)
160+ }
161+ entries = append(entries, entry)
162+ }
163+ normalized, err := normalizeCatalogCacheEntries(entries)
164+ if err != nil {
165+ return nil, err
166+ }
167+ return normalized, nil
168+}
169+
170+func migrateExactCatalogCacheV2(legacy catalogCacheFileV2) (catalogCacheFile, error) {
171+ if legacy.Version == nil || *legacy.Version != catalogCacheExactFileVersion || legacy.Hosts == nil {
172+ return catalogCacheFile{}, fmt.Errorf("remote catalog cache: malformed version 2 cache file")
173+ }
174+ hosts := make([]catalogCacheHost, 0, len(*legacy.Hosts))
175+ for _, host := range *legacy.Hosts {
176+ if host.Sessions == nil {
177+ return catalogCacheFile{}, fmt.Errorf("remote catalog cache: malformed cache file: missing host fields")
178+ }
179+ sessions := make([]catalogCacheSession, 0, len(*host.Sessions))
180+ for _, session := range *host.Sessions {
181+ if session.Tabs != nil || session.TabList == nil {
182+ return catalogCacheFile{}, fmt.Errorf("remote catalog cache: unsupported ambiguous version 2 tab count")
183+ }
184+ sessions = append(sessions, catalogCacheSession{
185+ LifecycleID: session.LifecycleID,
186+ Name: session.Name,
187+ State: session.State,
188+ Ephemeral: session.Ephemeral,
189+ LastUsedSeq: session.LastUsedSeq,
190+ Tabs: session.TabList,
191+ ActiveTabID: session.ActiveTabID,
192+ Attached: session.Attached,
193+ })
194+ }
195+ hosts = append(hosts, catalogCacheHost{
196+ Target: host.Target, FetchedAtUnixNano: host.FetchedAtUnixNano, Sessions: &sessions,
197+ })
198+ }
199+ version := catalogCacheFileVersion
200+ return catalogCacheFile{Version: &version, Hosts: &hosts}, nil
201+}
202+
203+func normalizeCatalogCacheEntries(entries []catalogue.RemoteCatalogCacheEntry) ([]catalogue.RemoteCatalogCacheEntry, error) {
204+ if err := catalogue.ValidateRemoteCatalogCacheEntries(entries); err != nil {
205+ return nil, err
206+ }
207+ normalized := make([]catalogue.RemoteCatalogCacheEntry, 0, len(entries))
208+ hosts := make(map[string]struct{}, len(entries))
209+ for _, entry := range entries {
210+ if err := domain.ValidateRemoteHostTarget(entry.Host); err != nil {
211+ return nil, fmt.Errorf("remote catalog cache: invalid host %q: %w", entry.Host, err)
212+ }
213+ if entry.FetchedAt.UnixNano() <= 0 {
214+ return nil, fmt.Errorf("remote catalog cache: host %q has non-positive fetched time", entry.Host)
215+ }
216+ if _, duplicate := hosts[entry.Host]; duplicate {
217+ return nil, fmt.Errorf("remote catalog cache: duplicate host %q", entry.Host)
218+ }
219+ hosts[entry.Host] = struct{}{}
220+
221+ copyEntry := catalogue.RemoteCatalogCacheEntry{
222+ Host: entry.Host,
223+ FetchedAt: entry.FetchedAt,
224+ Incarnation: entry.Incarnation,
225+ Sessions: make([]catalogue.RemoteCatalogSession, 0, len(entry.Sessions)),
226+ }
227+ sessions := make(map[string]struct{}, len(entry.Sessions))
228+ for _, session := range entry.Sessions {
229+ if !utf8.ValidString(session.Name) || !utf8.ValidString(string(session.State)) {
230+ return nil, fmt.Errorf("remote catalog cache: session is not valid UTF-8")
231+ }
232+ if err := domain.ValidateSessionName(session.Name); err != nil {
233+ return nil, fmt.Errorf("remote catalog cache: invalid session %q: %w", session.Name, err)
234+ }
235+ if !session.State.Valid() {
236+ return nil, fmt.Errorf("remote catalog cache: invalid session state %q", session.State)
237+ }
238+ if _, duplicate := sessions[session.Name]; duplicate {
239+ return nil, fmt.Errorf("remote catalog cache: duplicate session %q for host %q", session.Name, entry.Host)
240+ }
241+ sessions[session.Name] = struct{}{}
242+ copySession := session
243+ copySession.Tabs = make([]catalogue.RemoteCatalogTab, len(session.Tabs))
244+ for i, tab := range session.Tabs {
245+ copySession.Tabs[i] = catalogue.RemoteCatalogTab{ID: tab.ID, Index: tab.Index, Name: tab.Name}
246+ }
247+ copyEntry.Sessions = append(copyEntry.Sessions, copySession)
248+ }
249+ sort.Slice(copyEntry.Sessions, func(i, j int) bool {
250+ return copyEntry.Sessions[i].Name < copyEntry.Sessions[j].Name
251+ })
252+ normalized = append(normalized, copyEntry)
253+ }
254+ sort.Slice(normalized, func(i, j int) bool {
255+ return normalized[i].Host < normalized[j].Host
256+ })
257+ return normalized, nil
258+}
A internal/adapters/brokerstore/legacy_test.go +84 -0
@@ -0,0 +1,84 @@
1+package brokerstore
2+
3+import (
4+ "os"
5+ "path/filepath"
6+ "strings"
7+ "testing"
8+
9+ "github.com/stretchr/testify/require"
10+)
11+
12+func TestCacheStrictFailures(t *testing.T) {
13+ raw, err := os.ReadFile("testdata/cache-v4.json")
14+ require.NoError(t, err)
15+ for name, data := range map[string]string{
16+ "unknown": strings.Replace(string(raw), `"target":`, `"surprise":1,"target":`, 1),
17+ "duplicate": strings.Replace(string(raw), `"version": 4`, `"version":4,"version":4`, 1),
18+ "trailing": string(raw) + `{}`,
19+ "missing incarnation": strings.Replace(string(raw), `, "incarnation": "01000000000000000000000000000000"`, "", 1),
20+ "bad lifecycle": strings.Replace(string(raw), "02000000000000000000000000000000", "00000000000000000000000000000000", 1),
21+ "bad time": strings.Replace(string(raw), "1780000000000000000", "0", 1),
22+ "ambiguous v2": `{"version":2,"hosts":[{"target":"user@arch","fetched_at_unix_nano":1,"sessions":[{"lifecycle_id":"02000000000000000000000000000000","name":"work","state":"up","ephemeral":false,"tabs":1,"attached":false}]}]}`,
23+ "too many hosts": `{"version":4,"hosts":[` + strings.Repeat(`{"target":"user@arch","fetched_at_unix_nano":1,"incarnation":"01000000000000000000000000000000","sessions":[]},`, 64) + `{"target":"user@arch","fetched_at_unix_nano":1,"incarnation":"01000000000000000000000000000000","sessions":[]}]}`,
24+ } {
25+ t.Run(name, func(t *testing.T) {
26+ o := options(t)
27+ o.LegacyCache = filepath.Join(t.TempDir(), "cache")
28+ require.NoError(t, os.WriteFile(o.LegacyCache, []byte(data), 0600))
29+ _, err := OpenOffline(o)
30+ require.Error(t, err)
31+ })
32+ }
33+}
34+
35+func TestRecoveryCorruption(t *testing.T) {
36+ o := options(t)
37+ s, err := OpenOffline(o)
38+ require.NoError(t, err)
39+ require.NoError(t, s.Close())
40+ require.NoError(t, os.Remove(filepath.Join(o.Dir, "state.json")))
41+ raw, err := readBounded(filepath.Join(o.Dir, "recovery.json"))
42+ require.NoError(t, err)
43+ var r recovery
44+ require.NoError(t, strict(raw, &r))
45+ r.Hosts = []byte("changed")
46+ // Use the production atomic writer without changing its digest metadata.
47+ writer := Store{dir: o.Dir}
48+ require.NoError(t, writer.write("recovery.json", r))
49+ _, err = OpenOffline(o)
50+ require.ErrorContains(t, err, "digest mismatch")
51+}
52+
53+func TestDefensiveCopiesAndConcurrentAccess(t *testing.T) {
54+ o := options(t)
55+ s, err := OpenOffline(o)
56+ require.NoError(t, err)
57+ defer s.Close()
58+ h, err := s.LoadHosts()
59+ require.NoError(t, err)
60+ h.Hosts[0].Policy.Trust = "changed"
61+ snap, err := s.Load()
62+ require.NoError(t, err)
63+ snap.Hosts[0].Sessions[0].Tabs[0].Name = "changed"
64+ again, err := s.Load()
65+ require.NoError(t, err)
66+ require.Equal(t, "shell", again.Hosts[0].Sessions[0].Tabs[0].Name)
67+ hosts, err := s.LoadHosts()
68+ require.NoError(t, err)
69+ require.Equal(t, "known-hosts", hosts.Hosts[0].Policy.Trust)
70+ done := make(chan struct{})
71+ go func() {
72+ defer close(done)
73+ for i := 0; i < 30; i++ {
74+ _, _ = s.Load()
75+ _, _ = s.LoadHosts()
76+ }
77+ }()
78+ for i := 0; i < 30; i++ {
79+ hosts, err = s.LoadHosts()
80+ require.NoError(t, err)
81+ require.NoError(t, s.ReplaceHosts(hosts.Revision, hosts.Hosts))
82+ }
83+ <-done
84+}
@@ -0,0 +1,84 @@
1+package brokerstore
2+
3+import (
4+ "os"
5+ "path/filepath"
6+ "strings"
7+ "testing"
8+
9+ "github.com/stretchr/testify/require"
10+)
11+
12+func TestCacheStrictFailures(t *testing.T) {
13+ raw, err := os.ReadFile("testdata/cache-v4.json")
14+ require.NoError(t, err)
15+ for name, data := range map[string]string{
16+ "unknown": strings.Replace(string(raw), `"target":`, `"surprise":1,"target":`, 1),
17+ "duplicate": strings.Replace(string(raw), `"version": 4`, `"version":4,"version":4`, 1),
18+ "trailing": string(raw) + `{}`,
19+ "missing incarnation": strings.Replace(string(raw), `, "incarnation": "01000000000000000000000000000000"`, "", 1),
20+ "bad lifecycle": strings.Replace(string(raw), "02000000000000000000000000000000", "00000000000000000000000000000000", 1),
21+ "bad time": strings.Replace(string(raw), "1780000000000000000", "0", 1),
22+ "ambiguous v2": `{"version":2,"hosts":[{"target":"user@arch","fetched_at_unix_nano":1,"sessions":[{"lifecycle_id":"02000000000000000000000000000000","name":"work","state":"up","ephemeral":false,"tabs":1,"attached":false}]}]}`,
23+ "too many hosts": `{"version":4,"hosts":[` + strings.Repeat(`{"target":"user@arch","fetched_at_unix_nano":1,"incarnation":"01000000000000000000000000000000","sessions":[]},`, 64) + `{"target":"user@arch","fetched_at_unix_nano":1,"incarnation":"01000000000000000000000000000000","sessions":[]}]}`,
24+ } {
25+ t.Run(name, func(t *testing.T) {
26+ o := options(t)
27+ o.LegacyCache = filepath.Join(t.TempDir(), "cache")
28+ require.NoError(t, os.WriteFile(o.LegacyCache, []byte(data), 0600))
29+ _, err := OpenOffline(o)
30+ require.Error(t, err)
31+ })
32+ }
33+}
34+
35+func TestRecoveryCorruption(t *testing.T) {
36+ o := options(t)
37+ s, err := OpenOffline(o)
38+ require.NoError(t, err)
39+ require.NoError(t, s.Close())
40+ require.NoError(t, os.Remove(filepath.Join(o.Dir, "state.json")))
41+ raw, err := readBounded(filepath.Join(o.Dir, "recovery.json"))
42+ require.NoError(t, err)
43+ var r recovery
44+ require.NoError(t, strict(raw, &r))
45+ r.Hosts = []byte("changed")
46+ // Use the production atomic writer without changing its digest metadata.
47+ writer := Store{dir: o.Dir}
48+ require.NoError(t, writer.write("recovery.json", r))
49+ _, err = OpenOffline(o)
50+ require.ErrorContains(t, err, "digest mismatch")
51+}
52+
53+func TestDefensiveCopiesAndConcurrentAccess(t *testing.T) {
54+ o := options(t)
55+ s, err := OpenOffline(o)
56+ require.NoError(t, err)
57+ defer s.Close()
58+ h, err := s.LoadHosts()
59+ require.NoError(t, err)
60+ h.Hosts[0].Policy.Trust = "changed"
61+ snap, err := s.Load()
62+ require.NoError(t, err)
63+ snap.Hosts[0].Sessions[0].Tabs[0].Name = "changed"
64+ again, err := s.Load()
65+ require.NoError(t, err)
66+ require.Equal(t, "shell", again.Hosts[0].Sessions[0].Tabs[0].Name)
67+ hosts, err := s.LoadHosts()
68+ require.NoError(t, err)
69+ require.Equal(t, "known-hosts", hosts.Hosts[0].Policy.Trust)
70+ done := make(chan struct{})
71+ go func() {
72+ defer close(done)
73+ for i := 0; i < 30; i++ {
74+ _, _ = s.Load()
75+ _, _ = s.LoadHosts()
76+ }
77+ }()
78+ for i := 0; i < 30; i++ {
79+ hosts, err = s.LoadHosts()
80+ require.NoError(t, err)
81+ require.NoError(t, s.ReplaceHosts(hosts.Revision, hosts.Hosts))
82+ }
83+ <-done
84+}
A internal/adapters/brokerstore/store.go +455 -0
@@ -0,0 +1,455 @@
1+// Package brokerstore provides offline-only broker persistence. OpenOffline is
2+// never composed into application wiring. Legacy sources are immutable inputs:
3+// the lifetime lock is the same hosts.json.lock the legacy host writer takes,
4+// so a live host writer is excluded, but the legacy catalog cache source has no
5+// lock at all and must be immutable or quiesced. No environment policy is
6+// inferred.
7+package brokerstore
8+
9+import (
10+ "crypto/sha256"
11+ "encoding/hex"
12+ "encoding/json"
13+ "errors"
14+ "fmt"
15+ "os"
16+ "path/filepath"
17+ "sync"
18+ "syscall"
19+
20+ "github.com/bnema/vev/internal/ports"
21+ "github.com/bnema/vev/pkg/safedir"
22+)
23+
24+const MaxFileBytes = 16 << 20
25+
26+// lockFileName is the lifetime lock of a durable broker store. The name is
27+// deliberately the one the legacy host writer takes beside hosts.json, so a
28+// store opened on a directory that writer also uses excludes it for the
29+// store's lifetime. The legacy catalog cache source is written without any
30+// lock, so that source must be immutable or quiesced before migration.
31+const lockFileName = "hosts.json.lock"
32+
33+// ErrLocked reports that another owner already holds the lifetime lock.
34+var ErrLocked = fmt.Errorf("brokerstore: already owned: %w", ports.ErrBrokerStoreLocked)
35+
36+// ErrStale reports that the caller's authority token no longer matches durable
37+// state, or that the supplied membership would regress or re-trust an existing
38+// registration without a fresh identity.
39+var ErrStale = fmt.Errorf("brokerstore: stale authority: %w", ports.ErrBrokerHostConflict)
40+
41+// ErrInvalidState reports durable state that fails validation. The caller must
42+// resolve it offline from the recovery record; the store never replaces it.
43+var ErrInvalidState = fmt.Errorf("brokerstore: invalid state: %w", ports.ErrBrokerStoreInvalidState)
44+
45+// invalid wraps one durable-state validation failure.
46+func invalid(format string, args ...any) error {
47+ return fmt.Errorf("%w: %s", ErrInvalidState, fmt.Sprintf(format, args...))
48+}
49+
50+// manifestSourceLabel names one immutable recovery input by its generic role:
51+// the membership source that seeds authority, and the observation source that
52+// only seeds advisory inventory. A label never names a legacy file or schema.
53+type manifestSourceLabel string
54+
55+const (
56+ manifestMembership manifestSourceLabel = "membership"
57+ manifestObservations manifestSourceLabel = "observations"
58+)
59+
60+// manifestSource records one recovery input. Present distinguishes a source
61+// that was supplied but carried no records from a source that was never
62+// supplied: absence and emptiness can share a digest, so only presence makes
63+// the recovery record an exact description of the migration input.
64+type manifestSource struct {
65+ Label manifestSourceLabel
66+ Present bool
67+ SHA256 string
68+}
69+
70+type manifest struct {
71+ Version int
72+ Sources []manifestSource
73+}
74+
75+// source returns the manifest entry for one generic role label.
76+func (m manifest) source(label manifestSourceLabel) (manifestSource, bool) {
77+ for _, source := range m.Sources {
78+ if source.Label == label {
79+ return source, true
80+ }
81+ }
82+ return manifestSource{}, false
83+}
84+
85+// Options names a private offline destination and immutable legacy inputs.
86+// Policies must explicitly cover every imported endpoint. Fault is a test seam
87+// called after each durable-write boundary; an error poisons this open handle.
88+type Options struct {
89+ Dir, LegacyHosts, LegacyCache string
90+ Policies map[string]ports.BrokerPolicy
91+ Fault func(string) error
92+}
93+
94+type state struct {
95+ Manifest manifest
96+ Hosts ports.BrokerHosts
97+ Snapshot ports.BrokerSnapshot
98+}
99+type recovery struct {
100+ State state
101+ Hosts, Cache []byte
102+}
103+type Store struct {
104+ mu sync.Mutex
105+ dir string
106+ lock *os.File
107+ state state
108+ fault func(string) error
109+ failed bool
110+ writeEpoch ports.BrokerEpoch // first publication binds this lifetime handle
111+}
112+
113+var _ ports.BrokerHostStore = (*Store)(nil)
114+
115+// OpenOffline returns only after validated unified state is durable. The
116+// recovery record is written first and retained forever as the rollback copy;
117+// restart reuses its generated identities instead of migrating a second time.
118+// An existing corrupt state is an error, never silently replaced by a backup.
119+func OpenOffline(o Options) (*Store, error) {
120+ if err := safedir.EnsurePrivate(o.Dir); err != nil {
121+ return nil, err
122+ }
123+ fd, err := syscall.Open(filepath.Join(o.Dir, lockFileName), syscall.O_CREAT|syscall.O_RDWR|syscall.O_CLOEXEC|syscall.O_NOFOLLOW, 0600)
124+ if err != nil {
125+ return nil, err
126+ }
127+ lock := os.NewFile(uintptr(fd), lockFileName)
128+ info, err := lock.Stat()
129+ if err != nil {
130+ lock.Close()
131+ return nil, err
132+ }
133+ if !info.Mode().IsRegular() || info.Mode().Perm() != 0600 {
134+ lock.Close()
135+ return nil, invalid("unsafe owner lock")
136+ }
137+ if err = syscall.Flock(fd, syscall.LOCK_EX|syscall.LOCK_NB); err != nil {
138+ lock.Close()
139+ return nil, fmt.Errorf("%w: %v", ErrLocked, err)
140+ }
141+ s := &Store{dir: o.Dir, lock: lock, fault: o.Fault}
142+ ok := false
143+ defer func() {
144+ if !ok {
145+ s.Close()
146+ }
147+ }()
148+ raw, err := readBounded(filepath.Join(o.Dir, "state.json"))
149+ if err == nil {
150+ if err = strict(raw, &s.state); err != nil {
151+ return nil, invalid("state.json: %v", err)
152+ }
153+ } else if errors.Is(err, os.ErrNotExist) {
154+ var r recovery
155+ raw, err = readBounded(filepath.Join(o.Dir, "recovery.json"))
156+ if err == nil {
157+ if err = strict(raw, &r); err != nil {
158+ return nil, invalid("recovery.json: %v", err)
159+ }
160+ if err = r.verify(); err != nil {
161+ return nil, err
162+ }
163+ } else if errors.Is(err, os.ErrNotExist) {
164+ r, err = migrate(o)
165+ if err == nil {
166+ err = s.write("recovery.json", r)
167+ }
168+ } else {
169+ return nil, invalid("recovery.json: %v", err)
170+ }
171+ if err == nil {
172+ err = validate(r.State)
173+ }
174+ if err == nil {
175+ err = s.write("state.json", r.State)
176+ }
177+ s.state = r.State
178+ } else {
179+ return nil, invalid("state.json: %v", err)
180+ }
181+ if err != nil {
182+ return nil, err
183+ }
184+ if err = validate(s.state); err != nil {
185+ return nil, err
186+ }
187+ ok = true
188+ return s, nil
189+}
190+
191+// verify reports whether the recovery record's retained bytes match their
192+// manifest entries, including each source's recorded presence: an absent
193+// source must retain no bytes, and a supplied source keeps its exact digest.
194+func (r recovery) verify() error {
195+ membership, ok := r.State.Manifest.source(manifestMembership)
196+ if !ok || digest(r.Hosts) != membership.SHA256 {
197+ return invalid("recovery digest mismatch")
198+ }
199+ observations, ok := r.State.Manifest.source(manifestObservations)
200+ if !ok || digest(r.Cache) != observations.SHA256 {
201+ return invalid("recovery digest mismatch")
202+ }
203+ for _, source := range []struct {
204+ label manifestSourceLabel
205+ present bool
206+ bytes int
207+ }{{manifestMembership, membership.Present, len(r.Hosts)}, {manifestObservations, observations.Present, len(r.Cache)}} {
208+ if !source.present && source.bytes != 0 {
209+ return invalid("recovery records an absent %s source with retained bytes", source.label)
210+ }
211+ }
212+ return nil
213+}
214+func digest(b []byte) string { sum := sha256.Sum256(b); return hex.EncodeToString(sum[:]) }
215+func (s *Store) ready() error {
216+ if s.lock == nil || s.failed {
217+ return errors.New("brokerstore: closed or outcome unknown; reopen required")
218+ }
219+ return nil
220+}
221+func (s *Store) Close() error {
222+ s.mu.Lock()
223+ defer s.mu.Unlock()
224+ if s.lock == nil {
225+ return nil
226+ }
227+ err := s.lock.Close()
228+ s.lock = nil
229+ return err
230+}
231+func (s *Store) LoadHosts() (ports.BrokerHosts, error) {
232+ s.mu.Lock()
233+ defer s.mu.Unlock()
234+ if err := s.ready(); err != nil {
235+ return ports.BrokerHosts{}, err
236+ }
237+ h := s.state.Hosts
238+ h.Hosts = append([]ports.BrokerHostRecord(nil), h.Hosts...)
239+ return h, nil
240+}
241+func (s *Store) Load() (ports.BrokerSnapshot, error) {
242+ s.mu.Lock()
243+ defer s.mu.Unlock()
244+ if err := s.ready(); err != nil {
245+ return ports.BrokerSnapshot{}, err
246+ }
247+ return s.state.Snapshot.Clone(), nil
248+}
249+func (s *Store) ReplaceHosts(expected uint64, hosts []ports.BrokerHostRecord) error {
250+ s.mu.Lock()
251+ defer s.mu.Unlock()
252+ if err := s.ready(); err != nil {
253+ return err
254+ }
255+ // Validate caller membership before the CAS: malformed registration,
256+ // policy, anchoring, bound, or duplicate-endpoint input is a caller error,
257+ // not corrupt durable state, so it never surfaces as ErrInvalidState.
258+ if err := ports.ValidateBrokerHostRecords(hosts); err != nil {
259+ return err
260+ }
261+ if expected != s.state.Hosts.Revision || expected == ^uint64(0) {
262+ return ErrStale
263+ }
264+ next := s.state
265+ next.Hosts = ports.BrokerHosts{Revision: expected + 1, Hosts: append([]ports.BrokerHostRecord(nil), hosts...)}
266+ // Policy changes require a new registration generation as well; otherwise
267+ // an already queued observation could be accepted under changed trust.
268+ for _, h := range hosts {
269+ for _, old := range s.state.Hosts.Hosts {
270+ if h.Registration.Endpoint == old.Registration.Endpoint && h.Policy != old.Policy && h.Registration.Equal(old.Registration) {
271+ return ErrStale
272+ }
273+ if h.Registration.Incarnation == old.Registration.Incarnation && h.Registration.Endpoint == old.Registration.Endpoint && h.Registration.Generation < old.Registration.Generation {
274+ return ErrStale
275+ }
276+ }
277+ }
278+ next.Snapshot = filter(next.Snapshot, next.Hosts)
279+ return s.commit(next)
280+}
281+func (s *Store) Store(snapshot ports.BrokerSnapshot) error {
282+ s.mu.Lock()
283+ defer s.mu.Unlock()
284+ if err := s.ready(); err != nil {
285+ return err
286+ }
287+ // Removed tombstones and in-flight checking/failure detail are process-local
288+ // fencing and observation state; the durable snapshot never carries them, so
289+ // sanitize the caller's copy before validating the durable shape.
290+ snapshot = snapshot.Clone()
291+ snapshot.Removed = nil
292+ for i := range snapshot.Hosts {
293+ snapshot.Hosts[i].Checking = false
294+ snapshot.Hosts[i].LastFailure.Err = nil
295+ }
296+ if err := snapshot.Validate(); err != nil {
297+ return err
298+ }
299+ if s.writeEpoch != 0 && s.writeEpoch != snapshot.Epoch {
300+ return ErrStale
301+ }
302+ old := s.state.Snapshot
303+ if old.Epoch == snapshot.Epoch && old.Revision >= snapshot.Revision {
304+ return ErrStale
305+ }
306+ // Reject rather than partially adopt a queued publication from retired hosts.
307+ for _, h := range snapshot.Hosts {
308+ found := false
309+ for _, a := range s.state.Hosts.Hosts {
310+ if h.Registration.Equal(a.Registration) {
311+ found = true
312+ }
313+ }
314+ if !found {
315+ return ErrStale
316+ }
317+ }
318+ next := s.state
319+ next.Snapshot = snapshot.Clone()
320+ if err := s.commit(next); err != nil {
321+ return err
322+ }
323+ s.writeEpoch = snapshot.Epoch
324+ return nil
325+}
326+func filter(snapshot ports.BrokerSnapshot, hosts ports.BrokerHosts) ports.BrokerSnapshot {
327+ out := snapshot.Clone()
328+ out.Hosts = nil
329+ out.Removed = nil
330+ for _, h := range snapshot.Hosts {
331+ for _, a := range hosts.Hosts {
332+ if h.Registration.Equal(a.Registration) {
333+ out.Hosts = append(out.Hosts, h.Clone())
334+ }
335+ }
336+ }
337+ return out
338+}
339+func validate(st state) error {
340+ if st.Manifest.Version != 1 {
341+ return invalid("unsupported manifest")
342+ }
343+ labels := make(map[manifestSourceLabel]bool, len(st.Manifest.Sources))
344+ for _, source := range st.Manifest.Sources {
345+ if labels[source.Label] {
346+ return invalid("duplicate manifest source")
347+ }
348+ labels[source.Label] = true
349+ b, err := hex.DecodeString(source.SHA256)
350+ if err != nil || len(b) != 32 {
351+ return invalid("invalid digest")
352+ }
353+ }
354+ for _, label := range []manifestSourceLabel{manifestMembership, manifestObservations} {
355+ if !labels[label] {
356+ return invalid("missing manifest source")
357+ }
358+ }
359+ if err := st.Hosts.Validate(); err != nil {
360+ return invalid("%v", err)
361+ }
362+ if st.Snapshot.Epoch == 0 {
363+ if st.Snapshot.Revision != 0 || len(st.Snapshot.Hosts) != 0 || len(st.Snapshot.Removed) != 0 {
364+ return invalid("invalid empty snapshot")
365+ }
366+ return nil
367+ }
368+ if err := st.Snapshot.Validate(); err != nil {
369+ return invalid("%v", err)
370+ }
371+ for _, h := range st.Snapshot.Hosts {
372+ found := false
373+ for _, a := range st.Hosts.Hosts {
374+ if h.Registration.Equal(a.Registration) {
375+ found = true
376+ }
377+ }
378+ if !found || h.Checking || h.LastFailure.Err != nil {
379+ return invalid("non-authoritative snapshot")
380+ }
381+ // The store enforces the same durable projection rules the registry
382+ // checks before publishing, so an accepted publication always persists.
383+ if err := ports.ValidateDurableHostProjection(h); err != nil {
384+ return invalid("%v", err)
385+ }
386+ }
387+ return nil
388+}
389+func (s *Store) commit(next state) error {
390+ if err := validate(next); err != nil {
391+ return err
392+ }
393+ if err := s.write("state.json", next); err != nil {
394+ s.failed = true
395+ return err
396+ }
397+ s.state = next
398+ return nil
399+}
400+func (s *Store) boundary(name string) error {
401+ if s.fault != nil {
402+ return s.fault(name)
403+ }
404+ return nil
405+}
406+func (s *Store) write(name string, value any) error {
407+ data, err := json.Marshal(value)
408+ if err != nil {
409+ return err
410+ }
411+ if len(data) > MaxFileBytes {
412+ return errors.New("brokerstore: file too large")
413+ }
414+ f, err := os.CreateTemp(s.dir, ".pending-")
415+ if err != nil {
416+ return err
417+ }
418+ defer os.Remove(f.Name())
419+ defer f.Close()
420+ // The staged state carries rollback inputs and registration identities, so
421+ // its permissions are set explicitly instead of relying on the umask.
422+ if err = f.Chmod(0o600); err != nil {
423+ return err
424+ }
425+ if _, err = f.Write(data); err != nil {
426+ return err
427+ }
428+ if err = s.boundary(name + ":write"); err != nil {
429+ return err
430+ }
431+ if err = f.Sync(); err != nil {
432+ return err
433+ }
434+ if err = s.boundary(name + ":sync"); err != nil {
435+ return err
436+ }
437+ if err = f.Close(); err != nil {
438+ return err
439+ }
440+ if err = os.Rename(f.Name(), filepath.Join(s.dir, name)); err != nil {
441+ return err
442+ }
443+ if err = s.boundary(name + ":rename"); err != nil {
444+ return err
445+ }
446+ d, err := os.Open(s.dir)
447+ if err != nil {
448+ return err
449+ }
450+ defer d.Close()
451+ if err = d.Sync(); err != nil {
452+ return err
453+ }
454+ return s.boundary(name + ":dirsync")
455+}
@@ -0,0 +1,455 @@
1+// Package brokerstore provides offline-only broker persistence. OpenOffline is
2+// never composed into application wiring. Legacy sources are immutable inputs:
3+// the lifetime lock is the same hosts.json.lock the legacy host writer takes,
4+// so a live host writer is excluded, but the legacy catalog cache source has no
5+// lock at all and must be immutable or quiesced. No environment policy is
6+// inferred.
7+package brokerstore
8+
9+import (
10+ "crypto/sha256"
11+ "encoding/hex"
12+ "encoding/json"
13+ "errors"
14+ "fmt"
15+ "os"
16+ "path/filepath"
17+ "sync"
18+ "syscall"
19+
20+ "github.com/bnema/vev/internal/ports"
21+ "github.com/bnema/vev/pkg/safedir"
22+)
23+
24+const MaxFileBytes = 16 << 20
25+
26+// lockFileName is the lifetime lock of a durable broker store. The name is
27+// deliberately the one the legacy host writer takes beside hosts.json, so a
28+// store opened on a directory that writer also uses excludes it for the
29+// store's lifetime. The legacy catalog cache source is written without any
30+// lock, so that source must be immutable or quiesced before migration.
31+const lockFileName = "hosts.json.lock"
32+
33+// ErrLocked reports that another owner already holds the lifetime lock.
34+var ErrLocked = fmt.Errorf("brokerstore: already owned: %w", ports.ErrBrokerStoreLocked)
35+
36+// ErrStale reports that the caller's authority token no longer matches durable
37+// state, or that the supplied membership would regress or re-trust an existing
38+// registration without a fresh identity.
39+var ErrStale = fmt.Errorf("brokerstore: stale authority: %w", ports.ErrBrokerHostConflict)
40+
41+// ErrInvalidState reports durable state that fails validation. The caller must
42+// resolve it offline from the recovery record; the store never replaces it.
43+var ErrInvalidState = fmt.Errorf("brokerstore: invalid state: %w", ports.ErrBrokerStoreInvalidState)
44+
45+// invalid wraps one durable-state validation failure.
46+func invalid(format string, args ...any) error {
47+ return fmt.Errorf("%w: %s", ErrInvalidState, fmt.Sprintf(format, args...))
48+}
49+
50+// manifestSourceLabel names one immutable recovery input by its generic role:
51+// the membership source that seeds authority, and the observation source that
52+// only seeds advisory inventory. A label never names a legacy file or schema.
53+type manifestSourceLabel string
54+
55+const (
56+ manifestMembership manifestSourceLabel = "membership"
57+ manifestObservations manifestSourceLabel = "observations"
58+)
59+
60+// manifestSource records one recovery input. Present distinguishes a source
61+// that was supplied but carried no records from a source that was never
62+// supplied: absence and emptiness can share a digest, so only presence makes
63+// the recovery record an exact description of the migration input.
64+type manifestSource struct {
65+ Label manifestSourceLabel
66+ Present bool
67+ SHA256 string
68+}
69+
70+type manifest struct {
71+ Version int
72+ Sources []manifestSource
73+}
74+
75+// source returns the manifest entry for one generic role label.
76+func (m manifest) source(label manifestSourceLabel) (manifestSource, bool) {
77+ for _, source := range m.Sources {
78+ if source.Label == label {
79+ return source, true
80+ }
81+ }
82+ return manifestSource{}, false
83+}
84+
85+// Options names a private offline destination and immutable legacy inputs.
86+// Policies must explicitly cover every imported endpoint. Fault is a test seam
87+// called after each durable-write boundary; an error poisons this open handle.
88+type Options struct {
89+ Dir, LegacyHosts, LegacyCache string
90+ Policies map[string]ports.BrokerPolicy
91+ Fault func(string) error
92+}
93+
94+type state struct {
95+ Manifest manifest
96+ Hosts ports.BrokerHosts
97+ Snapshot ports.BrokerSnapshot
98+}
99+type recovery struct {
100+ State state
101+ Hosts, Cache []byte
102+}
103+type Store struct {
104+ mu sync.Mutex
105+ dir string
106+ lock *os.File
107+ state state
108+ fault func(string) error
109+ failed bool
110+ writeEpoch ports.BrokerEpoch // first publication binds this lifetime handle
111+}
112+
113+var _ ports.BrokerHostStore = (*Store)(nil)
114+
115+// OpenOffline returns only after validated unified state is durable. The
116+// recovery record is written first and retained forever as the rollback copy;
117+// restart reuses its generated identities instead of migrating a second time.
118+// An existing corrupt state is an error, never silently replaced by a backup.
119+func OpenOffline(o Options) (*Store, error) {
120+ if err := safedir.EnsurePrivate(o.Dir); err != nil {
121+ return nil, err
122+ }
123+ fd, err := syscall.Open(filepath.Join(o.Dir, lockFileName), syscall.O_CREAT|syscall.O_RDWR|syscall.O_CLOEXEC|syscall.O_NOFOLLOW, 0600)
124+ if err != nil {
125+ return nil, err
126+ }
127+ lock := os.NewFile(uintptr(fd), lockFileName)
128+ info, err := lock.Stat()
129+ if err != nil {
130+ lock.Close()
131+ return nil, err
132+ }
133+ if !info.Mode().IsRegular() || info.Mode().Perm() != 0600 {
134+ lock.Close()
135+ return nil, invalid("unsafe owner lock")
136+ }
137+ if err = syscall.Flock(fd, syscall.LOCK_EX|syscall.LOCK_NB); err != nil {
138+ lock.Close()
139+ return nil, fmt.Errorf("%w: %v", ErrLocked, err)
140+ }
141+ s := &Store{dir: o.Dir, lock: lock, fault: o.Fault}
142+ ok := false
143+ defer func() {
144+ if !ok {
145+ s.Close()
146+ }
147+ }()
148+ raw, err := readBounded(filepath.Join(o.Dir, "state.json"))
149+ if err == nil {
150+ if err = strict(raw, &s.state); err != nil {
151+ return nil, invalid("state.json: %v", err)
152+ }
153+ } else if errors.Is(err, os.ErrNotExist) {
154+ var r recovery
155+ raw, err = readBounded(filepath.Join(o.Dir, "recovery.json"))
156+ if err == nil {
157+ if err = strict(raw, &r); err != nil {
158+ return nil, invalid("recovery.json: %v", err)
159+ }
160+ if err = r.verify(); err != nil {
161+ return nil, err
162+ }
163+ } else if errors.Is(err, os.ErrNotExist) {
164+ r, err = migrate(o)
165+ if err == nil {
166+ err = s.write("recovery.json", r)
167+ }
168+ } else {
169+ return nil, invalid("recovery.json: %v", err)
170+ }
171+ if err == nil {
172+ err = validate(r.State)
173+ }
174+ if err == nil {
175+ err = s.write("state.json", r.State)
176+ }
177+ s.state = r.State
178+ } else {
179+ return nil, invalid("state.json: %v", err)
180+ }
181+ if err != nil {
182+ return nil, err
183+ }
184+ if err = validate(s.state); err != nil {
185+ return nil, err
186+ }
187+ ok = true
188+ return s, nil
189+}
190+
191+// verify reports whether the recovery record's retained bytes match their
192+// manifest entries, including each source's recorded presence: an absent
193+// source must retain no bytes, and a supplied source keeps its exact digest.
194+func (r recovery) verify() error {
195+ membership, ok := r.State.Manifest.source(manifestMembership)
196+ if !ok || digest(r.Hosts) != membership.SHA256 {
197+ return invalid("recovery digest mismatch")
198+ }
199+ observations, ok := r.State.Manifest.source(manifestObservations)
200+ if !ok || digest(r.Cache) != observations.SHA256 {
201+ return invalid("recovery digest mismatch")
202+ }
203+ for _, source := range []struct {
204+ label manifestSourceLabel
205+ present bool
206+ bytes int
207+ }{{manifestMembership, membership.Present, len(r.Hosts)}, {manifestObservations, observations.Present, len(r.Cache)}} {
208+ if !source.present && source.bytes != 0 {
209+ return invalid("recovery records an absent %s source with retained bytes", source.label)
210+ }
211+ }
212+ return nil
213+}
214+func digest(b []byte) string { sum := sha256.Sum256(b); return hex.EncodeToString(sum[:]) }
215+func (s *Store) ready() error {
216+ if s.lock == nil || s.failed {
217+ return errors.New("brokerstore: closed or outcome unknown; reopen required")
218+ }
219+ return nil
220+}
221+func (s *Store) Close() error {
222+ s.mu.Lock()
223+ defer s.mu.Unlock()
224+ if s.lock == nil {
225+ return nil
226+ }
227+ err := s.lock.Close()
228+ s.lock = nil
229+ return err
230+}
231+func (s *Store) LoadHosts() (ports.BrokerHosts, error) {
232+ s.mu.Lock()
233+ defer s.mu.Unlock()
234+ if err := s.ready(); err != nil {
235+ return ports.BrokerHosts{}, err
236+ }
237+ h := s.state.Hosts
238+ h.Hosts = append([]ports.BrokerHostRecord(nil), h.Hosts...)
239+ return h, nil
240+}
241+func (s *Store) Load() (ports.BrokerSnapshot, error) {
242+ s.mu.Lock()
243+ defer s.mu.Unlock()
244+ if err := s.ready(); err != nil {
245+ return ports.BrokerSnapshot{}, err
246+ }
247+ return s.state.Snapshot.Clone(), nil
248+}
249+func (s *Store) ReplaceHosts(expected uint64, hosts []ports.BrokerHostRecord) error {
250+ s.mu.Lock()
251+ defer s.mu.Unlock()
252+ if err := s.ready(); err != nil {
253+ return err
254+ }
255+ // Validate caller membership before the CAS: malformed registration,
256+ // policy, anchoring, bound, or duplicate-endpoint input is a caller error,
257+ // not corrupt durable state, so it never surfaces as ErrInvalidState.
258+ if err := ports.ValidateBrokerHostRecords(hosts); err != nil {
259+ return err
260+ }
261+ if expected != s.state.Hosts.Revision || expected == ^uint64(0) {
262+ return ErrStale
263+ }
264+ next := s.state
265+ next.Hosts = ports.BrokerHosts{Revision: expected + 1, Hosts: append([]ports.BrokerHostRecord(nil), hosts...)}
266+ // Policy changes require a new registration generation as well; otherwise
267+ // an already queued observation could be accepted under changed trust.
268+ for _, h := range hosts {
269+ for _, old := range s.state.Hosts.Hosts {
270+ if h.Registration.Endpoint == old.Registration.Endpoint && h.Policy != old.Policy && h.Registration.Equal(old.Registration) {
271+ return ErrStale
272+ }
273+ if h.Registration.Incarnation == old.Registration.Incarnation && h.Registration.Endpoint == old.Registration.Endpoint && h.Registration.Generation < old.Registration.Generation {
274+ return ErrStale
275+ }
276+ }
277+ }
278+ next.Snapshot = filter(next.Snapshot, next.Hosts)
279+ return s.commit(next)
280+}
281+func (s *Store) Store(snapshot ports.BrokerSnapshot) error {
282+ s.mu.Lock()
283+ defer s.mu.Unlock()
284+ if err := s.ready(); err != nil {
285+ return err
286+ }
287+ // Removed tombstones and in-flight checking/failure detail are process-local
288+ // fencing and observation state; the durable snapshot never carries them, so
289+ // sanitize the caller's copy before validating the durable shape.
290+ snapshot = snapshot.Clone()
291+ snapshot.Removed = nil
292+ for i := range snapshot.Hosts {
293+ snapshot.Hosts[i].Checking = false
294+ snapshot.Hosts[i].LastFailure.Err = nil
295+ }
296+ if err := snapshot.Validate(); err != nil {
297+ return err
298+ }
299+ if s.writeEpoch != 0 && s.writeEpoch != snapshot.Epoch {
300+ return ErrStale
301+ }
302+ old := s.state.Snapshot
303+ if old.Epoch == snapshot.Epoch && old.Revision >= snapshot.Revision {
304+ return ErrStale
305+ }
306+ // Reject rather than partially adopt a queued publication from retired hosts.
307+ for _, h := range snapshot.Hosts {
308+ found := false
309+ for _, a := range s.state.Hosts.Hosts {
310+ if h.Registration.Equal(a.Registration) {
311+ found = true
312+ }
313+ }
314+ if !found {
315+ return ErrStale
316+ }
317+ }
318+ next := s.state
319+ next.Snapshot = snapshot.Clone()
320+ if err := s.commit(next); err != nil {
321+ return err
322+ }
323+ s.writeEpoch = snapshot.Epoch
324+ return nil
325+}
326+func filter(snapshot ports.BrokerSnapshot, hosts ports.BrokerHosts) ports.BrokerSnapshot {
327+ out := snapshot.Clone()
328+ out.Hosts = nil
329+ out.Removed = nil
330+ for _, h := range snapshot.Hosts {
331+ for _, a := range hosts.Hosts {
332+ if h.Registration.Equal(a.Registration) {
333+ out.Hosts = append(out.Hosts, h.Clone())
334+ }
335+ }
336+ }
337+ return out
338+}
339+func validate(st state) error {
340+ if st.Manifest.Version != 1 {
341+ return invalid("unsupported manifest")
342+ }
343+ labels := make(map[manifestSourceLabel]bool, len(st.Manifest.Sources))
344+ for _, source := range st.Manifest.Sources {
345+ if labels[source.Label] {
346+ return invalid("duplicate manifest source")
347+ }
348+ labels[source.Label] = true
349+ b, err := hex.DecodeString(source.SHA256)
350+ if err != nil || len(b) != 32 {
351+ return invalid("invalid digest")
352+ }
353+ }
354+ for _, label := range []manifestSourceLabel{manifestMembership, manifestObservations} {
355+ if !labels[label] {
356+ return invalid("missing manifest source")
357+ }
358+ }
359+ if err := st.Hosts.Validate(); err != nil {
360+ return invalid("%v", err)
361+ }
362+ if st.Snapshot.Epoch == 0 {
363+ if st.Snapshot.Revision != 0 || len(st.Snapshot.Hosts) != 0 || len(st.Snapshot.Removed) != 0 {
364+ return invalid("invalid empty snapshot")
365+ }
366+ return nil
367+ }
368+ if err := st.Snapshot.Validate(); err != nil {
369+ return invalid("%v", err)
370+ }
371+ for _, h := range st.Snapshot.Hosts {
372+ found := false
373+ for _, a := range st.Hosts.Hosts {
374+ if h.Registration.Equal(a.Registration) {
375+ found = true
376+ }
377+ }
378+ if !found || h.Checking || h.LastFailure.Err != nil {
379+ return invalid("non-authoritative snapshot")
380+ }
381+ // The store enforces the same durable projection rules the registry
382+ // checks before publishing, so an accepted publication always persists.
383+ if err := ports.ValidateDurableHostProjection(h); err != nil {
384+ return invalid("%v", err)
385+ }
386+ }
387+ return nil
388+}
389+func (s *Store) commit(next state) error {
390+ if err := validate(next); err != nil {
391+ return err
392+ }
393+ if err := s.write("state.json", next); err != nil {
394+ s.failed = true
395+ return err
396+ }
397+ s.state = next
398+ return nil
399+}
400+func (s *Store) boundary(name string) error {
401+ if s.fault != nil {
402+ return s.fault(name)
403+ }
404+ return nil
405+}
406+func (s *Store) write(name string, value any) error {
407+ data, err := json.Marshal(value)
408+ if err != nil {
409+ return err
410+ }
411+ if len(data) > MaxFileBytes {
412+ return errors.New("brokerstore: file too large")
413+ }
414+ f, err := os.CreateTemp(s.dir, ".pending-")
415+ if err != nil {
416+ return err
417+ }
418+ defer os.Remove(f.Name())
419+ defer f.Close()
420+ // The staged state carries rollback inputs and registration identities, so
421+ // its permissions are set explicitly instead of relying on the umask.
422+ if err = f.Chmod(0o600); err != nil {
423+ return err
424+ }
425+ if _, err = f.Write(data); err != nil {
426+ return err
427+ }
428+ if err = s.boundary(name + ":write"); err != nil {
429+ return err
430+ }
431+ if err = f.Sync(); err != nil {
432+ return err
433+ }
434+ if err = s.boundary(name + ":sync"); err != nil {
435+ return err
436+ }
437+ if err = f.Close(); err != nil {
438+ return err
439+ }
440+ if err = os.Rename(f.Name(), filepath.Join(s.dir, name)); err != nil {
441+ return err
442+ }
443+ if err = s.boundary(name + ":rename"); err != nil {
444+ return err
445+ }
446+ d, err := os.Open(s.dir)
447+ if err != nil {
448+ return err
449+ }
450+ defer d.Close()
451+ if err = d.Sync(); err != nil {
452+ return err
453+ }
454+ return s.boundary(name + ":dirsync")
455+}
A internal/adapters/brokerstore/store_test.go +628 -0
@@ -0,0 +1,628 @@
1+package brokerstore
2+
3+import (
4+ "errors"
5+ "fmt"
6+ "os"
7+ "os/exec"
8+ "path/filepath"
9+ "strings"
10+ "sync"
11+ "syscall"
12+ "testing"
13+
14+ "github.com/bnema/vev/internal/domain"
15+ "github.com/bnema/vev/internal/ports"
16+ "github.com/bnema/vev/internal/protocol"
17+ "github.com/stretchr/testify/require"
18+)
19+
20+func options(t *testing.T) Options {
21+ t.Helper()
22+ return Options{Dir: filepath.Join(t.TempDir(), "broker"), LegacyHosts: "testdata/hosts-v3.json", LegacyCache: "testdata/cache-v4.json", Policies: map[string]ports.BrokerPolicy{"user@arch": {ProtocolVersion: 1, CatalogSchemaVersion: 1, EnvironmentPolicy: protocol.EnvironmentPolicyDaemonOwned, Transport: "quic", Trust: "known-hosts", Launch: "explicit", Isolation: "user"}}}
23+}
24+func TestPriorFixtures(t *testing.T) {
25+ for h := 1; h <= 3; h++ {
26+ for c := 2; c <= 4; c++ {
27+ t.Run(fmt.Sprintf("hosts%d-cache%d", h, c), func(t *testing.T) {
28+ o := options(t)
29+ o.LegacyHosts = fmt.Sprintf("testdata/hosts-v%d.json", h)
30+ o.LegacyCache = fmt.Sprintf("testdata/cache-v%d.json", c)
31+ beforeH, err := os.ReadFile(o.LegacyHosts)
32+ require.NoError(t, err)
33+ beforeC, err := os.ReadFile(o.LegacyCache)
34+ require.NoError(t, err)
35+ s, err := OpenOffline(o)
36+ require.NoError(t, err)
37+ hosts, err := s.LoadHosts()
38+ require.NoError(t, err)
39+ require.Len(t, hosts.Hosts, 1)
40+ require.True(t, hosts.Hosts[0].Learned)
41+ require.Equal(t, h != 1, hosts.Hosts[0].Pinned)
42+ require.Equal(t, o.Policies["user@arch"], hosts.Hosts[0].Policy)
43+ if h == 3 {
44+ require.Equal(t, domain.RemoteGeneration(7), hosts.Hosts[0].Registration.Generation)
45+ }
46+ snap, err := s.Load()
47+ require.NoError(t, err)
48+ if h == 3 && c == 4 {
49+ require.Len(t, snap.Hosts, 1)
50+ require.Len(t, snap.Hosts[0].Sessions, 1)
51+ require.Equal(t, "t_work", snap.Hosts[0].Sessions[0].Tabs[0].ID)
52+ } else {
53+ require.Empty(t, snap.Hosts)
54+ }
55+ require.NoError(t, s.Close())
56+ o.LegacyHosts = "missing"
57+ o.LegacyCache = "missing"
58+ o.Policies = nil
59+ s, err = OpenOffline(o)
60+ require.NoError(t, err)
61+ defer s.Close()
62+ again, err := s.LoadHosts()
63+ require.NoError(t, err)
64+ require.Equal(t, hosts, again)
65+ raw, err := readBounded(filepath.Join(o.Dir, "recovery.json"))
66+ require.NoError(t, err)
67+ var r recovery
68+ require.NoError(t, strict(raw, &r))
69+ require.Equal(t, beforeH, r.Hosts)
70+ require.Equal(t, beforeC, r.Cache)
71+ for _, name := range []string{"state.json", "recovery.json", lockFileName} {
72+ st, err := os.Stat(filepath.Join(o.Dir, name))
73+ require.NoError(t, err)
74+ require.Equal(t, os.FileMode(0600), st.Mode().Perm())
75+ }
76+ })
77+ }
78+ }
79+}
80+func TestMigrationFaultRestart(t *testing.T) {
81+ for _, file := range []string{"recovery.json", "state.json"} {
82+ for _, point := range []string{"write", "sync", "rename", "dirsync"} {
83+ t.Run(file+point, func(t *testing.T) {
84+ o := options(t)
85+ o.LegacyHosts = "testdata/hosts-v2.json"
86+ fault := errors.New("power loss")
87+ o.Fault = func(p string) error {
88+ if p == file+":"+point {
89+ return fault
90+ }
91+ return nil
92+ }
93+ s, err := OpenOffline(o)
94+ require.ErrorIs(t, err, fault)
95+ require.Nil(t, s)
96+ var saved recovery
97+ raw, e := readBounded(filepath.Join(o.Dir, "recovery.json"))
98+ if e == nil {
99+ require.NoError(t, strict(raw, &saved))
100+ }
101+ o.Fault = nil
102+ s, err = OpenOffline(o)
103+ require.NoError(t, err)
104+ defer s.Close()
105+ hosts, err := s.LoadHosts()
106+ require.NoError(t, err)
107+ if e == nil {
108+ require.Equal(t, saved.State.Hosts, hosts)
109+ }
110+ })
111+ }
112+ }
113+}
114+func TestLockLifetime(t *testing.T) {
115+ if dir := os.Getenv("VEV_BROKERSTORE_LOCK_TEST"); dir != "" {
116+ s, err := OpenOffline(Options{Dir: dir})
117+ if s != nil {
118+ s.Close()
119+ }
120+ if !errors.Is(err, ErrLocked) {
121+ os.Exit(2)
122+ }
123+ return
124+ }
125+ o := options(t)
126+ s, err := OpenOffline(o)
127+ require.NoError(t, err)
128+ // The lifetime lock carries the legacy host writer's exact file name, so a
129+ // store opened beside a live hosts.json excludes that writer too.
130+ _, err = os.Stat(filepath.Join(o.Dir, "hosts.json.lock"))
131+ require.NoError(t, err)
132+ cmd := exec.Command(os.Args[0], "-test.run=^TestLockLifetime$")
133+ cmd.Env = append(os.Environ(), "VEV_BROKERSTORE_LOCK_TEST="+o.Dir)
134+ require.NoError(t, cmd.Run())
135+ // Assertions never run inside the competing goroutines: t.FailNow from a
136+ // non-test goroutine is invalid, so each worker records its outcome and the
137+ // test body checks them after the join.
138+ var wg sync.WaitGroup
139+ outcomes := make([]error, 8)
140+ for i := 0; i < len(outcomes); i++ {
141+ wg.Add(1)
142+ go func(i int) {
143+ defer wg.Done()
144+ other, e := OpenOffline(o)
145+ if other != nil {
146+ _ = other.Close()
147+ outcomes[i] = fmt.Errorf("concurrent OpenOffline returned a store: %w", ports.ErrBrokerStoreLocked)
148+ return
149+ }
150+ if !errors.Is(e, ErrLocked) || !errors.Is(e, ports.ErrBrokerStoreLocked) {
151+ outcomes[i] = fmt.Errorf("concurrent OpenOffline error = %v, want ErrLocked", e)
152+ }
153+ }(i)
154+ }
155+ wg.Wait()
156+ for i, outcome := range outcomes {
157+ require.NoError(t, outcome, "worker %d", i)
158+ }
159+ require.NoError(t, s.Close())
160+ _, err = s.Load()
161+ require.Error(t, err)
162+ s, err = OpenOffline(o)
163+ require.NoError(t, err)
164+ require.NoError(t, s.Close())
165+}
166+
167+// TestLockExcludesLegacyHostWriter holds the lock the way the legacy host
168+// writer does and proves the store refuses to own the same directory.
169+func TestLockExcludesLegacyHostWriter(t *testing.T) {
170+ o := options(t)
171+ s, err := OpenOffline(o)
172+ require.NoError(t, err)
173+ require.NoError(t, s.Close())
174+
175+ legacyLock, err := os.OpenFile(filepath.Join(o.Dir, "hosts.json.lock"), os.O_CREATE|os.O_RDWR, 0600)
176+ require.NoError(t, err)
177+ defer legacyLock.Close()
178+ require.NoError(t, syscall.Flock(int(legacyLock.Fd()), syscall.LOCK_EX|syscall.LOCK_NB))
179+
180+ _, err = OpenOffline(o)
181+ require.ErrorIs(t, err, ErrLocked)
182+ require.ErrorIs(t, err, ports.ErrBrokerStoreLocked)
183+
184+ require.NoError(t, syscall.Flock(int(legacyLock.Fd()), syscall.LOCK_UN))
185+ s, err = OpenOffline(o)
186+ require.NoError(t, err)
187+ require.NoError(t, s.Close())
188+}
189+func TestStrictCorruption(t *testing.T) {
190+ for _, raw := range []string{`{"version":1,"hosts":[]}`, `{"version":3,"hosts":[]}`} {
191+ var v any
192+ require.NoError(t, strict([]byte(raw), &v))
193+ }
194+ for name, raw := range map[string]string{"duplicate": `{"version":1,"version":1,"hosts":[]}`, "unknown": `{"version":1,"hosts":[],"extra":0}`, "trailing": `{"version":1,"hosts":[]} {}`, "missing": `{"version":2,"pinned":[]}`, "future": `{"version":99,"hosts":[]}`, "utf8": string([]byte{255}), "oversize": strings.Repeat(" ", MaxFileBytes+1), "empty": ""} {
195+ t.Run(name, func(t *testing.T) {
196+ o := options(t)
197+ o.LegacyHosts = filepath.Join(t.TempDir(), "hosts")
198+ require.NoError(t, os.WriteFile(o.LegacyHosts, []byte(raw), 0600))
199+ s, err := OpenOffline(o)
200+ require.Error(t, err)
201+ require.Nil(t, s)
202+ _, err = os.Stat(filepath.Join(o.Dir, "state.json"))
203+ require.True(t, os.IsNotExist(err))
204+ })
205+ }
206+ t.Run("no implicit policy", func(t *testing.T) { o := options(t); o.Policies = nil; _, err := OpenOffline(o); require.Error(t, err) })
207+ t.Run("state never falls back", func(t *testing.T) {
208+ o := options(t)
209+ s, err := OpenOffline(o)
210+ require.NoError(t, err)
211+ require.NoError(t, s.Close())
212+ require.NoError(t, os.WriteFile(filepath.Join(o.Dir, "state.json"), []byte(`{`), 0600))
213+ _, err = OpenOffline(o)
214+ require.Error(t, err)
215+ })
216+ t.Run("source symlink", func(t *testing.T) {
217+ o := options(t)
218+ p := filepath.Join(t.TempDir(), "link")
219+ require.NoError(t, os.Symlink(o.LegacyHosts, p))
220+ o.LegacyHosts = p
221+ _, err := OpenOffline(o)
222+ require.Error(t, err)
223+ })
224+}
225+func TestDurableStateSentinelMapping(t *testing.T) {
226+ o := options(t)
227+ s, err := OpenOffline(o)
228+ require.NoError(t, err)
229+
230+ // A conflicting authority token is a host conflict, not a generic failure.
231+ require.ErrorIs(t, s.ReplaceHosts(0, nil), ErrStale)
232+ require.ErrorIs(t, s.ReplaceHosts(0, nil), ports.ErrBrokerHostConflict)
233+
234+ // A locked, corrupt, or closed store is classified without importing the
235+ // adapter: callers retry, resolve state offline, or reopen respectively.
236+ require.ErrorIs(t, ErrLocked, ports.ErrBrokerStoreLocked)
237+ require.ErrorIs(t, ErrInvalidState, ports.ErrBrokerStoreInvalidState)
238+
239+ require.NoError(t, s.Close())
240+ // Closing releases ownership without discarding the durable files.
241+ require.Error(t, s.ReplaceHosts(1, nil))
242+ require.NoError(t, os.WriteFile(filepath.Join(o.Dir, "state.json"), []byte(`{`), 0600))
243+ _, err = OpenOffline(o)
244+ require.ErrorIs(t, err, ErrInvalidState)
245+ require.ErrorIs(t, err, ports.ErrBrokerStoreInvalidState)
246+}
247+
248+// TestNonRegularInputsAreRejected covers inputs that are neither regular files
249+// nor absent: a directory and a FIFO must both fail closed before decoding.
250+func TestNonRegularInputsAreRejected(t *testing.T) {
251+ for _, name := range []string{"directory", "fifo"} {
252+ t.Run(name, func(t *testing.T) {
253+ o := options(t)
254+ path := filepath.Join(t.TempDir(), name)
255+ if name == "directory" {
256+ require.NoError(t, os.Mkdir(path, 0700))
257+ } else {
258+ require.NoError(t, syscall.Mkfifo(path, 0600))
259+ }
260+ o.LegacyHosts = path
261+ s, err := OpenOffline(o)
262+ require.Error(t, err)
263+ require.Nil(t, s)
264+ _, err = os.Stat(filepath.Join(o.Dir, "state.json"))
265+ require.True(t, os.IsNotExist(err))
266+ })
267+ }
268+}
269+
270+// TestCommittedStateAndRecoveryAreNeverFollowed replaces the committed state and
271+// the recovery record with symlinks to valid bytes: the store reads both with
272+// O_NOFOLLOW, so a symlink fails closed instead of redirecting a rename or
273+// silently adopting attacker-chosen state.
274+func TestCommittedStateAndRecoveryAreNeverFollowed(t *testing.T) {
275+ t.Run("state", func(t *testing.T) {
276+ o := options(t)
277+ s, err := OpenOffline(o)
278+ require.NoError(t, err)
279+ require.NoError(t, s.Close())
280+
281+ state := filepath.Join(o.Dir, "state.json")
282+ raw, err := os.ReadFile(state)
283+ require.NoError(t, err)
284+ target := filepath.Join(o.Dir, "elsewhere.json")
285+ require.NoError(t, os.WriteFile(target, raw, 0600))
286+ require.NoError(t, os.Remove(state))
287+ require.NoError(t, os.Symlink(target, state))
288+
289+ _, err = OpenOffline(o)
290+ require.ErrorIs(t, err, ErrInvalidState)
291+ require.ErrorIs(t, err, ports.ErrBrokerStoreInvalidState)
292+ after, err := os.ReadFile(target)
293+ require.NoError(t, err)
294+ require.Equal(t, raw, after)
295+ })
296+ t.Run("recovery", func(t *testing.T) {
297+ o := options(t)
298+ s, err := OpenOffline(o)
299+ require.NoError(t, err)
300+ require.NoError(t, s.Close())
301+
302+ recoveryPath := filepath.Join(o.Dir, "recovery.json")
303+ raw, err := os.ReadFile(recoveryPath)
304+ require.NoError(t, err)
305+ target := filepath.Join(o.Dir, "elsewhere.json")
306+ require.NoError(t, os.WriteFile(target, raw, 0600))
307+ require.NoError(t, os.Remove(recoveryPath))
308+ require.NoError(t, os.Remove(filepath.Join(o.Dir, "state.json")))
309+ require.NoError(t, os.Symlink(target, recoveryPath))
310+
311+ _, err = OpenOffline(o)
312+ require.ErrorIs(t, err, ErrInvalidState)
313+ require.ErrorIs(t, err, ports.ErrBrokerStoreInvalidState)
314+ _, err = os.Stat(filepath.Join(o.Dir, "state.json"))
315+ require.True(t, os.IsNotExist(err))
316+ })
317+}
318+
319+// TestManifestDistinguishesAbsentFromEmptySources pins the recovery manifest: an
320+// absent source and a supplied source that carries no records can produce the
321+// same membership, so presence, keyed by generic role label, is what makes the
322+// recovery record an exact description of the migration input.
323+func TestManifestDistinguishesAbsentFromEmptySources(t *testing.T) {
324+ readManifest := func(t *testing.T, dir string) (manifest, string) {
325+ t.Helper()
326+ raw, err := readBounded(filepath.Join(dir, "recovery.json"))
327+ require.NoError(t, err)
328+ var r recovery
329+ require.NoError(t, strict(raw, &r))
330+ return r.State.Manifest, digest(nil)
331+ }
332+
333+ absent := options(t)
334+ absent.LegacyHosts = ""
335+ absent.LegacyCache = ""
336+ absent.Policies = nil
337+ absentStore, err := OpenOffline(absent)
338+ require.NoError(t, err)
339+ defer absentStore.Close()
340+ hosts, err := absentStore.LoadHosts()
341+ require.NoError(t, err)
342+ require.Empty(t, hosts.Hosts)
343+ absentManifest, emptyDigest := readManifest(t, absent.Dir)
344+ for _, label := range []manifestSourceLabel{manifestMembership, manifestObservations} {
345+ source, ok := absentManifest.source(label)
346+ require.True(t, ok, label)
347+ require.False(t, source.Present, label)
348+ require.Equal(t, emptyDigest, source.SHA256, label)
349+ }
350+
351+ empty := options(t)
352+ empty.LegacyCache = ""
353+ empty.LegacyHosts = filepath.Join(t.TempDir(), "hosts")
354+ require.NoError(t, os.WriteFile(empty.LegacyHosts, []byte(`{"version":1,"hosts":[]}`), 0600))
355+ emptyStore, err := OpenOffline(empty)
356+ require.NoError(t, err)
357+ defer emptyStore.Close()
358+ stillEmpty, err := emptyStore.LoadHosts()
359+ require.NoError(t, err)
360+ // The same empty membership results, but the manifest records that the
361+ // membership source was supplied and what it contained.
362+ require.Equal(t, hosts, stillEmpty)
363+ emptyManifest, _ := readManifest(t, empty.Dir)
364+ membership, ok := emptyManifest.source(manifestMembership)
365+ require.True(t, ok)
366+ require.True(t, membership.Present)
367+ require.NotEqual(t, emptyDigest, membership.SHA256)
368+ observations, ok := emptyManifest.source(manifestObservations)
369+ require.True(t, ok)
370+ require.False(t, observations.Present)
371+
372+ // A record that claims the source was absent while retaining its bytes is
373+ // rejected: presence is verified, not merely recorded.
374+ raw, err := readBounded(filepath.Join(empty.Dir, "recovery.json"))
375+ require.NoError(t, err)
376+ var r recovery
377+ require.NoError(t, strict(raw, &r))
378+ for i := range r.State.Manifest.Sources {
379+ if r.State.Manifest.Sources[i].Label == manifestMembership {
380+ r.State.Manifest.Sources[i].Present = false
381+ }
382+ }
383+ writer := Store{dir: empty.Dir}
384+ require.NoError(t, writer.write("recovery.json", r))
385+ require.NoError(t, os.Remove(filepath.Join(empty.Dir, "state.json")))
386+ require.NoError(t, emptyStore.Close())
387+ _, err = OpenOffline(Options{Dir: empty.Dir})
388+ require.ErrorIs(t, err, ErrInvalidState)
389+ require.ErrorContains(t, err, "absent membership source")
390+}
391+
392+// TestRevisionOverflowIsRefused keeps the store from ever minting revision zero
393+// or accepting an exhausted authority token.
394+func TestRevisionOverflowIsRefused(t *testing.T) {
395+ o := options(t)
396+ s, err := OpenOffline(o)
397+ require.NoError(t, err)
398+ defer s.Close()
399+ h, err := s.LoadHosts()
400+ require.NoError(t, err)
401+ for _, hosts := range [][]ports.BrokerHostRecord{h.Hosts, nil} {
402+ require.ErrorIs(t, s.ReplaceHosts(^uint64(0), hosts), ErrStale)
403+ }
404+ after, err := s.LoadHosts()
405+ require.NoError(t, err)
406+ require.Equal(t, h, after)
407+}
408+
409+// TestAuthorityGenerationPolicyAndRemoveReAdd exercises the durable authority
410+// rules: policy changes need a fresh generation, generations never regress, a
411+// removed incarnation's observation never returns, and a re-add carries a new
412+// incarnation.
413+func TestAuthorityGenerationPolicyAndRemoveReAdd(t *testing.T) {
414+ o := options(t)
415+ s, err := OpenOffline(o)
416+ require.NoError(t, err)
417+ defer s.Close()
418+ h, err := s.LoadHosts()
419+ require.NoError(t, err)
420+ require.Len(t, h.Hosts, 1)
421+ base := h.Hosts[0]
422+
423+ // A policy change with an unchanged registration identity is refused: an
424+ // already queued observation must never be accepted under changed trust.
425+ changed := append([]ports.BrokerHostRecord(nil), h.Hosts...)
426+ changed[0].Policy.Trust = "changed"
427+ require.ErrorIs(t, s.ReplaceHosts(h.Revision, changed), ErrStale)
428+
429+ // Re-supplying identical membership is allowed and still advances the
430+ // authority revision.
431+ require.NoError(t, s.ReplaceHosts(h.Revision, h.Hosts))
432+ h2, err := s.LoadHosts()
433+ require.NoError(t, err)
434+ require.Equal(t, h.Revision+1, h2.Revision)
435+
436+ // The same policy change with a bumped generation is accepted and durable.
437+ changed[0].Registration.Generation++
438+ require.NoError(t, s.ReplaceHosts(h2.Revision, changed))
439+ h3, err := s.LoadHosts()
440+ require.NoError(t, err)
441+ require.Equal(t, "changed", h3.Hosts[0].Policy.Trust)
442+
443+ // A regressed generation on the same incarnation is refused.
444+ regressed := append([]ports.BrokerHostRecord(nil), h3.Hosts...)
445+ regressed[0].Registration.Generation = base.Registration.Generation
446+ require.ErrorIs(t, s.ReplaceHosts(h3.Revision, regressed), ErrStale)
447+
448+ // Full removal drops the observation with the registration.
449+ require.NoError(t, s.ReplaceHosts(h3.Revision, nil))
450+ removed, err := s.LoadHosts()
451+ require.NoError(t, err)
452+ require.Empty(t, removed.Hosts)
453+
454+ // A queued publication from the retired incarnation is refused even though
455+ // its revision is newer.
456+ require.ErrorIs(t, s.Store(ports.BrokerSnapshot{Epoch: 9, Revision: 1, Hosts: []ports.RemoteHostSnapshot{{Endpoint: base.Registration.Endpoint, Registration: base.Registration, Availability: domain.RemoteAvailabilityReachable}}}), ErrStale)
457+
458+ // Re-adding the endpoint needs a fresh incarnation; the retired
459+ // observation never comes back for it.
460+ var incarnation [16]byte
461+ incarnation[0] = 9
462+ registration, err := domain.NewRemoteRegistration(base.Registration.Endpoint, incarnation)
463+ require.NoError(t, err)
464+ reAdd := append([]ports.BrokerHostRecord(nil), base)
465+ reAdd[0].Registration = registration
466+ require.NoError(t, s.ReplaceHosts(removed.Revision, reAdd))
467+ readded, err := s.LoadHosts()
468+ require.NoError(t, err)
469+ require.True(t, readded.Hosts[0].Registration.Equal(registration))
470+ snapshot, err := s.Load()
471+ require.NoError(t, err)
472+ _, ok := snapshot.Find(base.Registration.Endpoint)
473+ require.False(t, ok, "a removed incarnation's observation must not survive a re-add")
474+}
475+
476+// TestReplaceHostsRejectsCallerInputBeforeCAS proves malformed caller
477+// membership is rejected as a caller error before the compare-and-swap, never
478+// as ErrBrokerStoreInvalidState: the durable state is a victim, not the cause.
479+func TestReplaceHostsRejectsCallerInputBeforeCAS(t *testing.T) {
480+ o := options(t)
481+ s, err := OpenOffline(o)
482+ require.NoError(t, err)
483+ defer s.Close()
484+ h, err := s.LoadHosts()
485+ require.NoError(t, err)
486+ require.Len(t, h.Hosts, 1)
487+ base := h.Hosts[0]
488+
489+ invalid := map[string][]ports.BrokerHostRecord{
490+ "zero registration": {{Pinned: true, Policy: base.Policy}},
491+ "zero policy": {{Registration: base.Registration, Pinned: true}},
492+ "unanchored": {{Registration: base.Registration, Policy: base.Policy}},
493+ "duplicate endpoint": {
494+ {Registration: base.Registration, Pinned: true, Policy: base.Policy},
495+ {Registration: base.Registration, Learned: true, Policy: base.Policy},
496+ },
497+ }
498+ for name, records := range invalid {
499+ t.Run(name, func(t *testing.T) {
500+ // The CAS token matches, so a CAS-first store would have accepted the
501+ // malformed set and failed later as invalid durable state.
502+ err := s.ReplaceHosts(h.Revision, records)
503+ require.Error(t, err)
504+ require.NotErrorIs(t, err, ErrStale)
505+ require.NotErrorIs(t, err, ErrInvalidState)
506+ require.NotErrorIs(t, err, ports.ErrBrokerStoreInvalidState)
507+ })
508+ }
509+ t.Run("over bound", func(t *testing.T) {
510+ records := make([]ports.BrokerHostRecord, 0, ports.BrokerMaxHosts+1)
511+ for index := 0; index <= ports.BrokerMaxHosts; index++ {
512+ registration, err := domain.NewRemoteRegistration(fmt.Sprintf("bound-%d.test", index), [16]byte{byte(index + 1)})
513+ require.NoError(t, err)
514+ records = append(records, ports.BrokerHostRecord{Registration: registration, Pinned: true, Policy: base.Policy})
515+ }
516+ err := s.ReplaceHosts(h.Revision, records)
517+ require.Error(t, err)
518+ require.NotErrorIs(t, err, ports.ErrBrokerStoreInvalidState)
519+ })
520+
521+ // Rejected input never commits: the authority revision and membership are
522+ // exactly as before.
523+ after, err := s.LoadHosts()
524+ require.NoError(t, err)
525+ require.Equal(t, h, after)
526+}
527+
528+// TestDurableSnapshotDropsProcessLocalTombstones proves a snapshot's
529+// process-local tombstone set is sanitized before persistence: storing a
530+// snapshot whose tombstone names a live host is accepted and reopened without
531+// Removed, instead of being rejected as durable-state corruption.
532+func TestDurableSnapshotDropsProcessLocalTombstones(t *testing.T) {
533+ o := options(t)
534+ s, err := OpenOffline(o)
535+ require.NoError(t, err)
536+ h, err := s.LoadHosts()
537+ require.NoError(t, err)
538+ require.Len(t, h.Hosts, 1)
539+ reg := h.Hosts[0].Registration
540+
541+ snap, err := s.Load()
542+ require.NoError(t, err)
543+ snap.Epoch = 5
544+ snap.Revision = 7
545+ snap.Removed = []ports.BrokerHostTombstone{{Endpoint: reg.Endpoint, Registration: reg, RetiredRevision: 6}}
546+ require.NoError(t, s.Store(snap))
547+
548+ stored, err := s.Load()
549+ require.NoError(t, err)
550+ require.Empty(t, stored.Removed)
551+ require.NoError(t, s.Close())
552+
553+ s, err = OpenOffline(o)
554+ require.NoError(t, err)
555+ defer s.Close()
556+ reopened, err := s.Load()
557+ require.NoError(t, err)
558+ require.Empty(t, reopened.Removed)
559+}
560+
561+func TestFencingAndCommitFaults(t *testing.T) {
562+ for _, point := range []string{"write", "sync", "rename", "dirsync"} {
563+ t.Run(point, func(t *testing.T) {
564+ o := options(t)
565+ s, err := OpenOffline(o)
566+ require.NoError(t, err)
567+ h, err := s.LoadHosts()
568+ require.NoError(t, err)
569+ snap, err := s.Load()
570+ require.NoError(t, err)
571+ snap.Epoch = 20
572+ snap.Revision = 1
573+ require.NoError(t, s.Store(snap))
574+ require.ErrorIs(t, s.Store(snap), ErrStale)
575+ other := snap
576+ other.Epoch = 21
577+ require.ErrorIs(t, s.Store(other), ErrStale)
578+ require.ErrorIs(t, s.ReplaceHosts(0, nil), ErrStale)
579+ changed := append([]ports.BrokerHostRecord(nil), h.Hosts...)
580+ changed[0].Policy.Trust = "new"
581+ require.ErrorIs(t, s.ReplaceHosts(h.Revision, changed), ErrStale)
582+ s.fault = func(p string) error {
583+ if p == "state.json:"+point {
584+ return errors.New("disk fault")
585+ }
586+ return nil
587+ }
588+ require.Error(t, s.ReplaceHosts(h.Revision, nil))
589+ _, err = s.LoadHosts()
590+ require.Error(t, err)
591+ require.NoError(t, s.Close())
592+ s, err = OpenOffline(o)
593+ require.NoError(t, err)
594+ defer s.Close()
595+ h, err = s.LoadHosts()
596+ require.NoError(t, err)
597+ if point == "rename" || point == "dirsync" {
598+ require.Empty(t, h.Hosts)
599+ snap.Revision++
600+ require.ErrorIs(t, s.Store(snap), ErrStale)
601+ } else {
602+ require.Len(t, h.Hosts, 1)
603+ }
604+ })
605+ }
606+ t.Run("generation and restart", func(t *testing.T) {
607+ o := options(t)
608+ s, err := OpenOffline(o)
609+ require.NoError(t, err)
610+ h, err := s.LoadHosts()
611+ require.NoError(t, err)
612+ snap, err := s.Load()
613+ require.NoError(t, err)
614+ h.Hosts[0].Registration.Generation++
615+ require.NoError(t, s.ReplaceHosts(h.Revision, h.Hosts))
616+ require.ErrorIs(t, s.Store(snap), ErrStale)
617+ require.NoError(t, s.Close())
618+ s, err = OpenOffline(o)
619+ require.NoError(t, err)
620+ defer s.Close()
621+ got, err := s.Load()
622+ require.NoError(t, err)
623+ require.Empty(t, got.Hosts)
624+ gotH, err := s.LoadHosts()
625+ require.NoError(t, err)
626+ require.Equal(t, h.Hosts, gotH.Hosts)
627+ })
628+}
@@ -0,0 +1,628 @@
1+package brokerstore
2+
3+import (
4+ "errors"
5+ "fmt"
6+ "os"
7+ "os/exec"
8+ "path/filepath"
9+ "strings"
10+ "sync"
11+ "syscall"
12+ "testing"
13+
14+ "github.com/bnema/vev/internal/domain"
15+ "github.com/bnema/vev/internal/ports"
16+ "github.com/bnema/vev/internal/protocol"
17+ "github.com/stretchr/testify/require"
18+)
19+
20+func options(t *testing.T) Options {
21+ t.Helper()
22+ return Options{Dir: filepath.Join(t.TempDir(), "broker"), LegacyHosts: "testdata/hosts-v3.json", LegacyCache: "testdata/cache-v4.json", Policies: map[string]ports.BrokerPolicy{"user@arch": {ProtocolVersion: 1, CatalogSchemaVersion: 1, EnvironmentPolicy: protocol.EnvironmentPolicyDaemonOwned, Transport: "quic", Trust: "known-hosts", Launch: "explicit", Isolation: "user"}}}
23+}
24+func TestPriorFixtures(t *testing.T) {
25+ for h := 1; h <= 3; h++ {
26+ for c := 2; c <= 4; c++ {
27+ t.Run(fmt.Sprintf("hosts%d-cache%d", h, c), func(t *testing.T) {
28+ o := options(t)
29+ o.LegacyHosts = fmt.Sprintf("testdata/hosts-v%d.json", h)
30+ o.LegacyCache = fmt.Sprintf("testdata/cache-v%d.json", c)
31+ beforeH, err := os.ReadFile(o.LegacyHosts)
32+ require.NoError(t, err)
33+ beforeC, err := os.ReadFile(o.LegacyCache)
34+ require.NoError(t, err)
35+ s, err := OpenOffline(o)
36+ require.NoError(t, err)
37+ hosts, err := s.LoadHosts()
38+ require.NoError(t, err)
39+ require.Len(t, hosts.Hosts, 1)
40+ require.True(t, hosts.Hosts[0].Learned)
41+ require.Equal(t, h != 1, hosts.Hosts[0].Pinned)
42+ require.Equal(t, o.Policies["user@arch"], hosts.Hosts[0].Policy)
43+ if h == 3 {
44+ require.Equal(t, domain.RemoteGeneration(7), hosts.Hosts[0].Registration.Generation)
45+ }
46+ snap, err := s.Load()
47+ require.NoError(t, err)
48+ if h == 3 && c == 4 {
49+ require.Len(t, snap.Hosts, 1)
50+ require.Len(t, snap.Hosts[0].Sessions, 1)
51+ require.Equal(t, "t_work", snap.Hosts[0].Sessions[0].Tabs[0].ID)
52+ } else {
53+ require.Empty(t, snap.Hosts)
54+ }
55+ require.NoError(t, s.Close())
56+ o.LegacyHosts = "missing"
57+ o.LegacyCache = "missing"
58+ o.Policies = nil
59+ s, err = OpenOffline(o)
60+ require.NoError(t, err)
61+ defer s.Close()
62+ again, err := s.LoadHosts()
63+ require.NoError(t, err)
64+ require.Equal(t, hosts, again)
65+ raw, err := readBounded(filepath.Join(o.Dir, "recovery.json"))
66+ require.NoError(t, err)
67+ var r recovery
68+ require.NoError(t, strict(raw, &r))
69+ require.Equal(t, beforeH, r.Hosts)
70+ require.Equal(t, beforeC, r.Cache)
71+ for _, name := range []string{"state.json", "recovery.json", lockFileName} {
72+ st, err := os.Stat(filepath.Join(o.Dir, name))
73+ require.NoError(t, err)
74+ require.Equal(t, os.FileMode(0600), st.Mode().Perm())
75+ }
76+ })
77+ }
78+ }
79+}
80+func TestMigrationFaultRestart(t *testing.T) {
81+ for _, file := range []string{"recovery.json", "state.json"} {
82+ for _, point := range []string{"write", "sync", "rename", "dirsync"} {
83+ t.Run(file+point, func(t *testing.T) {
84+ o := options(t)
85+ o.LegacyHosts = "testdata/hosts-v2.json"
86+ fault := errors.New("power loss")
87+ o.Fault = func(p string) error {
88+ if p == file+":"+point {
89+ return fault
90+ }
91+ return nil
92+ }
93+ s, err := OpenOffline(o)
94+ require.ErrorIs(t, err, fault)
95+ require.Nil(t, s)
96+ var saved recovery
97+ raw, e := readBounded(filepath.Join(o.Dir, "recovery.json"))
98+ if e == nil {
99+ require.NoError(t, strict(raw, &saved))
100+ }
101+ o.Fault = nil
102+ s, err = OpenOffline(o)
103+ require.NoError(t, err)
104+ defer s.Close()
105+ hosts, err := s.LoadHosts()
106+ require.NoError(t, err)
107+ if e == nil {
108+ require.Equal(t, saved.State.Hosts, hosts)
109+ }
110+ })
111+ }
112+ }
113+}
114+func TestLockLifetime(t *testing.T) {
115+ if dir := os.Getenv("VEV_BROKERSTORE_LOCK_TEST"); dir != "" {
116+ s, err := OpenOffline(Options{Dir: dir})
117+ if s != nil {
118+ s.Close()
119+ }
120+ if !errors.Is(err, ErrLocked) {
121+ os.Exit(2)
122+ }
123+ return
124+ }
125+ o := options(t)
126+ s, err := OpenOffline(o)
127+ require.NoError(t, err)
128+ // The lifetime lock carries the legacy host writer's exact file name, so a
129+ // store opened beside a live hosts.json excludes that writer too.
130+ _, err = os.Stat(filepath.Join(o.Dir, "hosts.json.lock"))
131+ require.NoError(t, err)
132+ cmd := exec.Command(os.Args[0], "-test.run=^TestLockLifetime$")
133+ cmd.Env = append(os.Environ(), "VEV_BROKERSTORE_LOCK_TEST="+o.Dir)
134+ require.NoError(t, cmd.Run())
135+ // Assertions never run inside the competing goroutines: t.FailNow from a
136+ // non-test goroutine is invalid, so each worker records its outcome and the
137+ // test body checks them after the join.
138+ var wg sync.WaitGroup
139+ outcomes := make([]error, 8)
140+ for i := 0; i < len(outcomes); i++ {
141+ wg.Add(1)
142+ go func(i int) {
143+ defer wg.Done()
144+ other, e := OpenOffline(o)
145+ if other != nil {
146+ _ = other.Close()
147+ outcomes[i] = fmt.Errorf("concurrent OpenOffline returned a store: %w", ports.ErrBrokerStoreLocked)
148+ return
149+ }
150+ if !errors.Is(e, ErrLocked) || !errors.Is(e, ports.ErrBrokerStoreLocked) {
151+ outcomes[i] = fmt.Errorf("concurrent OpenOffline error = %v, want ErrLocked", e)
152+ }
153+ }(i)
154+ }
155+ wg.Wait()
156+ for i, outcome := range outcomes {
157+ require.NoError(t, outcome, "worker %d", i)
158+ }
159+ require.NoError(t, s.Close())
160+ _, err = s.Load()
161+ require.Error(t, err)
162+ s, err = OpenOffline(o)
163+ require.NoError(t, err)
164+ require.NoError(t, s.Close())
165+}
166+
167+// TestLockExcludesLegacyHostWriter holds the lock the way the legacy host
168+// writer does and proves the store refuses to own the same directory.
169+func TestLockExcludesLegacyHostWriter(t *testing.T) {
170+ o := options(t)
171+ s, err := OpenOffline(o)
172+ require.NoError(t, err)
173+ require.NoError(t, s.Close())
174+
175+ legacyLock, err := os.OpenFile(filepath.Join(o.Dir, "hosts.json.lock"), os.O_CREATE|os.O_RDWR, 0600)
176+ require.NoError(t, err)
177+ defer legacyLock.Close()
178+ require.NoError(t, syscall.Flock(int(legacyLock.Fd()), syscall.LOCK_EX|syscall.LOCK_NB))
179+
180+ _, err = OpenOffline(o)
181+ require.ErrorIs(t, err, ErrLocked)
182+ require.ErrorIs(t, err, ports.ErrBrokerStoreLocked)
183+
184+ require.NoError(t, syscall.Flock(int(legacyLock.Fd()), syscall.LOCK_UN))
185+ s, err = OpenOffline(o)
186+ require.NoError(t, err)
187+ require.NoError(t, s.Close())
188+}
189+func TestStrictCorruption(t *testing.T) {
190+ for _, raw := range []string{`{"version":1,"hosts":[]}`, `{"version":3,"hosts":[]}`} {
191+ var v any
192+ require.NoError(t, strict([]byte(raw), &v))
193+ }
194+ for name, raw := range map[string]string{"duplicate": `{"version":1,"version":1,"hosts":[]}`, "unknown": `{"version":1,"hosts":[],"extra":0}`, "trailing": `{"version":1,"hosts":[]} {}`, "missing": `{"version":2,"pinned":[]}`, "future": `{"version":99,"hosts":[]}`, "utf8": string([]byte{255}), "oversize": strings.Repeat(" ", MaxFileBytes+1), "empty": ""} {
195+ t.Run(name, func(t *testing.T) {
196+ o := options(t)
197+ o.LegacyHosts = filepath.Join(t.TempDir(), "hosts")
198+ require.NoError(t, os.WriteFile(o.LegacyHosts, []byte(raw), 0600))
199+ s, err := OpenOffline(o)
200+ require.Error(t, err)
201+ require.Nil(t, s)
202+ _, err = os.Stat(filepath.Join(o.Dir, "state.json"))
203+ require.True(t, os.IsNotExist(err))
204+ })
205+ }
206+ t.Run("no implicit policy", func(t *testing.T) { o := options(t); o.Policies = nil; _, err := OpenOffline(o); require.Error(t, err) })
207+ t.Run("state never falls back", func(t *testing.T) {
208+ o := options(t)
209+ s, err := OpenOffline(o)
210+ require.NoError(t, err)
211+ require.NoError(t, s.Close())
212+ require.NoError(t, os.WriteFile(filepath.Join(o.Dir, "state.json"), []byte(`{`), 0600))
213+ _, err = OpenOffline(o)
214+ require.Error(t, err)
215+ })
216+ t.Run("source symlink", func(t *testing.T) {
217+ o := options(t)
218+ p := filepath.Join(t.TempDir(), "link")
219+ require.NoError(t, os.Symlink(o.LegacyHosts, p))
220+ o.LegacyHosts = p
221+ _, err := OpenOffline(o)
222+ require.Error(t, err)
223+ })
224+}
225+func TestDurableStateSentinelMapping(t *testing.T) {
226+ o := options(t)
227+ s, err := OpenOffline(o)
228+ require.NoError(t, err)
229+
230+ // A conflicting authority token is a host conflict, not a generic failure.
231+ require.ErrorIs(t, s.ReplaceHosts(0, nil), ErrStale)
232+ require.ErrorIs(t, s.ReplaceHosts(0, nil), ports.ErrBrokerHostConflict)
233+
234+ // A locked, corrupt, or closed store is classified without importing the
235+ // adapter: callers retry, resolve state offline, or reopen respectively.
236+ require.ErrorIs(t, ErrLocked, ports.ErrBrokerStoreLocked)
237+ require.ErrorIs(t, ErrInvalidState, ports.ErrBrokerStoreInvalidState)
238+
239+ require.NoError(t, s.Close())
240+ // Closing releases ownership without discarding the durable files.
241+ require.Error(t, s.ReplaceHosts(1, nil))
242+ require.NoError(t, os.WriteFile(filepath.Join(o.Dir, "state.json"), []byte(`{`), 0600))
243+ _, err = OpenOffline(o)
244+ require.ErrorIs(t, err, ErrInvalidState)
245+ require.ErrorIs(t, err, ports.ErrBrokerStoreInvalidState)
246+}
247+
248+// TestNonRegularInputsAreRejected covers inputs that are neither regular files
249+// nor absent: a directory and a FIFO must both fail closed before decoding.
250+func TestNonRegularInputsAreRejected(t *testing.T) {
251+ for _, name := range []string{"directory", "fifo"} {
252+ t.Run(name, func(t *testing.T) {
253+ o := options(t)
254+ path := filepath.Join(t.TempDir(), name)
255+ if name == "directory" {
256+ require.NoError(t, os.Mkdir(path, 0700))
257+ } else {
258+ require.NoError(t, syscall.Mkfifo(path, 0600))
259+ }
260+ o.LegacyHosts = path
261+ s, err := OpenOffline(o)
262+ require.Error(t, err)
263+ require.Nil(t, s)
264+ _, err = os.Stat(filepath.Join(o.Dir, "state.json"))
265+ require.True(t, os.IsNotExist(err))
266+ })
267+ }
268+}
269+
270+// TestCommittedStateAndRecoveryAreNeverFollowed replaces the committed state and
271+// the recovery record with symlinks to valid bytes: the store reads both with
272+// O_NOFOLLOW, so a symlink fails closed instead of redirecting a rename or
273+// silently adopting attacker-chosen state.
274+func TestCommittedStateAndRecoveryAreNeverFollowed(t *testing.T) {
275+ t.Run("state", func(t *testing.T) {
276+ o := options(t)
277+ s, err := OpenOffline(o)
278+ require.NoError(t, err)
279+ require.NoError(t, s.Close())
280+
281+ state := filepath.Join(o.Dir, "state.json")
282+ raw, err := os.ReadFile(state)
283+ require.NoError(t, err)
284+ target := filepath.Join(o.Dir, "elsewhere.json")
285+ require.NoError(t, os.WriteFile(target, raw, 0600))
286+ require.NoError(t, os.Remove(state))
287+ require.NoError(t, os.Symlink(target, state))
288+
289+ _, err = OpenOffline(o)
290+ require.ErrorIs(t, err, ErrInvalidState)
291+ require.ErrorIs(t, err, ports.ErrBrokerStoreInvalidState)
292+ after, err := os.ReadFile(target)
293+ require.NoError(t, err)
294+ require.Equal(t, raw, after)
295+ })
296+ t.Run("recovery", func(t *testing.T) {
297+ o := options(t)
298+ s, err := OpenOffline(o)
299+ require.NoError(t, err)
300+ require.NoError(t, s.Close())
301+
302+ recoveryPath := filepath.Join(o.Dir, "recovery.json")
303+ raw, err := os.ReadFile(recoveryPath)
304+ require.NoError(t, err)
305+ target := filepath.Join(o.Dir, "elsewhere.json")
306+ require.NoError(t, os.WriteFile(target, raw, 0600))
307+ require.NoError(t, os.Remove(recoveryPath))
308+ require.NoError(t, os.Remove(filepath.Join(o.Dir, "state.json")))
309+ require.NoError(t, os.Symlink(target, recoveryPath))
310+
311+ _, err = OpenOffline(o)
312+ require.ErrorIs(t, err, ErrInvalidState)
313+ require.ErrorIs(t, err, ports.ErrBrokerStoreInvalidState)
314+ _, err = os.Stat(filepath.Join(o.Dir, "state.json"))
315+ require.True(t, os.IsNotExist(err))
316+ })
317+}
318+
319+// TestManifestDistinguishesAbsentFromEmptySources pins the recovery manifest: an
320+// absent source and a supplied source that carries no records can produce the
321+// same membership, so presence, keyed by generic role label, is what makes the
322+// recovery record an exact description of the migration input.
323+func TestManifestDistinguishesAbsentFromEmptySources(t *testing.T) {
324+ readManifest := func(t *testing.T, dir string) (manifest, string) {
325+ t.Helper()
326+ raw, err := readBounded(filepath.Join(dir, "recovery.json"))
327+ require.NoError(t, err)
328+ var r recovery
329+ require.NoError(t, strict(raw, &r))
330+ return r.State.Manifest, digest(nil)
331+ }
332+
333+ absent := options(t)
334+ absent.LegacyHosts = ""
335+ absent.LegacyCache = ""
336+ absent.Policies = nil
337+ absentStore, err := OpenOffline(absent)
338+ require.NoError(t, err)
339+ defer absentStore.Close()
340+ hosts, err := absentStore.LoadHosts()
341+ require.NoError(t, err)
342+ require.Empty(t, hosts.Hosts)
343+ absentManifest, emptyDigest := readManifest(t, absent.Dir)
344+ for _, label := range []manifestSourceLabel{manifestMembership, manifestObservations} {
345+ source, ok := absentManifest.source(label)
346+ require.True(t, ok, label)
347+ require.False(t, source.Present, label)
348+ require.Equal(t, emptyDigest, source.SHA256, label)
349+ }
350+
351+ empty := options(t)
352+ empty.LegacyCache = ""
353+ empty.LegacyHosts = filepath.Join(t.TempDir(), "hosts")
354+ require.NoError(t, os.WriteFile(empty.LegacyHosts, []byte(`{"version":1,"hosts":[]}`), 0600))
355+ emptyStore, err := OpenOffline(empty)
356+ require.NoError(t, err)
357+ defer emptyStore.Close()
358+ stillEmpty, err := emptyStore.LoadHosts()
359+ require.NoError(t, err)
360+ // The same empty membership results, but the manifest records that the
361+ // membership source was supplied and what it contained.
362+ require.Equal(t, hosts, stillEmpty)
363+ emptyManifest, _ := readManifest(t, empty.Dir)
364+ membership, ok := emptyManifest.source(manifestMembership)
365+ require.True(t, ok)
366+ require.True(t, membership.Present)
367+ require.NotEqual(t, emptyDigest, membership.SHA256)
368+ observations, ok := emptyManifest.source(manifestObservations)
369+ require.True(t, ok)
370+ require.False(t, observations.Present)
371+
372+ // A record that claims the source was absent while retaining its bytes is
373+ // rejected: presence is verified, not merely recorded.
374+ raw, err := readBounded(filepath.Join(empty.Dir, "recovery.json"))
375+ require.NoError(t, err)
376+ var r recovery
377+ require.NoError(t, strict(raw, &r))
378+ for i := range r.State.Manifest.Sources {
379+ if r.State.Manifest.Sources[i].Label == manifestMembership {
380+ r.State.Manifest.Sources[i].Present = false
381+ }
382+ }
383+ writer := Store{dir: empty.Dir}
384+ require.NoError(t, writer.write("recovery.json", r))
385+ require.NoError(t, os.Remove(filepath.Join(empty.Dir, "state.json")))
386+ require.NoError(t, emptyStore.Close())
387+ _, err = OpenOffline(Options{Dir: empty.Dir})
388+ require.ErrorIs(t, err, ErrInvalidState)
389+ require.ErrorContains(t, err, "absent membership source")
390+}
391+
392+// TestRevisionOverflowIsRefused keeps the store from ever minting revision zero
393+// or accepting an exhausted authority token.
394+func TestRevisionOverflowIsRefused(t *testing.T) {
395+ o := options(t)
396+ s, err := OpenOffline(o)
397+ require.NoError(t, err)
398+ defer s.Close()
399+ h, err := s.LoadHosts()
400+ require.NoError(t, err)
401+ for _, hosts := range [][]ports.BrokerHostRecord{h.Hosts, nil} {
402+ require.ErrorIs(t, s.ReplaceHosts(^uint64(0), hosts), ErrStale)
403+ }
404+ after, err := s.LoadHosts()
405+ require.NoError(t, err)
406+ require.Equal(t, h, after)
407+}
408+
409+// TestAuthorityGenerationPolicyAndRemoveReAdd exercises the durable authority
410+// rules: policy changes need a fresh generation, generations never regress, a
411+// removed incarnation's observation never returns, and a re-add carries a new
412+// incarnation.
413+func TestAuthorityGenerationPolicyAndRemoveReAdd(t *testing.T) {
414+ o := options(t)
415+ s, err := OpenOffline(o)
416+ require.NoError(t, err)
417+ defer s.Close()
418+ h, err := s.LoadHosts()
419+ require.NoError(t, err)
420+ require.Len(t, h.Hosts, 1)
421+ base := h.Hosts[0]
422+
423+ // A policy change with an unchanged registration identity is refused: an
424+ // already queued observation must never be accepted under changed trust.
425+ changed := append([]ports.BrokerHostRecord(nil), h.Hosts...)
426+ changed[0].Policy.Trust = "changed"
427+ require.ErrorIs(t, s.ReplaceHosts(h.Revision, changed), ErrStale)
428+
429+ // Re-supplying identical membership is allowed and still advances the
430+ // authority revision.
431+ require.NoError(t, s.ReplaceHosts(h.Revision, h.Hosts))
432+ h2, err := s.LoadHosts()
433+ require.NoError(t, err)
434+ require.Equal(t, h.Revision+1, h2.Revision)
435+
436+ // The same policy change with a bumped generation is accepted and durable.
437+ changed[0].Registration.Generation++
438+ require.NoError(t, s.ReplaceHosts(h2.Revision, changed))
439+ h3, err := s.LoadHosts()
440+ require.NoError(t, err)
441+ require.Equal(t, "changed", h3.Hosts[0].Policy.Trust)
442+
443+ // A regressed generation on the same incarnation is refused.
444+ regressed := append([]ports.BrokerHostRecord(nil), h3.Hosts...)
445+ regressed[0].Registration.Generation = base.Registration.Generation
446+ require.ErrorIs(t, s.ReplaceHosts(h3.Revision, regressed), ErrStale)
447+
448+ // Full removal drops the observation with the registration.
449+ require.NoError(t, s.ReplaceHosts(h3.Revision, nil))
450+ removed, err := s.LoadHosts()
451+ require.NoError(t, err)
452+ require.Empty(t, removed.Hosts)
453+
454+ // A queued publication from the retired incarnation is refused even though
455+ // its revision is newer.
456+ require.ErrorIs(t, s.Store(ports.BrokerSnapshot{Epoch: 9, Revision: 1, Hosts: []ports.RemoteHostSnapshot{{Endpoint: base.Registration.Endpoint, Registration: base.Registration, Availability: domain.RemoteAvailabilityReachable}}}), ErrStale)
457+
458+ // Re-adding the endpoint needs a fresh incarnation; the retired
459+ // observation never comes back for it.
460+ var incarnation [16]byte
461+ incarnation[0] = 9
462+ registration, err := domain.NewRemoteRegistration(base.Registration.Endpoint, incarnation)
463+ require.NoError(t, err)
464+ reAdd := append([]ports.BrokerHostRecord(nil), base)
465+ reAdd[0].Registration = registration
466+ require.NoError(t, s.ReplaceHosts(removed.Revision, reAdd))
467+ readded, err := s.LoadHosts()
468+ require.NoError(t, err)
469+ require.True(t, readded.Hosts[0].Registration.Equal(registration))
470+ snapshot, err := s.Load()
471+ require.NoError(t, err)
472+ _, ok := snapshot.Find(base.Registration.Endpoint)
473+ require.False(t, ok, "a removed incarnation's observation must not survive a re-add")
474+}
475+
476+// TestReplaceHostsRejectsCallerInputBeforeCAS proves malformed caller
477+// membership is rejected as a caller error before the compare-and-swap, never
478+// as ErrBrokerStoreInvalidState: the durable state is a victim, not the cause.
479+func TestReplaceHostsRejectsCallerInputBeforeCAS(t *testing.T) {
480+ o := options(t)
481+ s, err := OpenOffline(o)
482+ require.NoError(t, err)
483+ defer s.Close()
484+ h, err := s.LoadHosts()
485+ require.NoError(t, err)
486+ require.Len(t, h.Hosts, 1)
487+ base := h.Hosts[0]
488+
489+ invalid := map[string][]ports.BrokerHostRecord{
490+ "zero registration": {{Pinned: true, Policy: base.Policy}},
491+ "zero policy": {{Registration: base.Registration, Pinned: true}},
492+ "unanchored": {{Registration: base.Registration, Policy: base.Policy}},
493+ "duplicate endpoint": {
494+ {Registration: base.Registration, Pinned: true, Policy: base.Policy},
495+ {Registration: base.Registration, Learned: true, Policy: base.Policy},
496+ },
497+ }
498+ for name, records := range invalid {
499+ t.Run(name, func(t *testing.T) {
500+ // The CAS token matches, so a CAS-first store would have accepted the
501+ // malformed set and failed later as invalid durable state.
502+ err := s.ReplaceHosts(h.Revision, records)
503+ require.Error(t, err)
504+ require.NotErrorIs(t, err, ErrStale)
505+ require.NotErrorIs(t, err, ErrInvalidState)
506+ require.NotErrorIs(t, err, ports.ErrBrokerStoreInvalidState)
507+ })
508+ }
509+ t.Run("over bound", func(t *testing.T) {
510+ records := make([]ports.BrokerHostRecord, 0, ports.BrokerMaxHosts+1)
511+ for index := 0; index <= ports.BrokerMaxHosts; index++ {
512+ registration, err := domain.NewRemoteRegistration(fmt.Sprintf("bound-%d.test", index), [16]byte{byte(index + 1)})
513+ require.NoError(t, err)
514+ records = append(records, ports.BrokerHostRecord{Registration: registration, Pinned: true, Policy: base.Policy})
515+ }
516+ err := s.ReplaceHosts(h.Revision, records)
517+ require.Error(t, err)
518+ require.NotErrorIs(t, err, ports.ErrBrokerStoreInvalidState)
519+ })
520+
521+ // Rejected input never commits: the authority revision and membership are
522+ // exactly as before.
523+ after, err := s.LoadHosts()
524+ require.NoError(t, err)
525+ require.Equal(t, h, after)
526+}
527+
528+// TestDurableSnapshotDropsProcessLocalTombstones proves a snapshot's
529+// process-local tombstone set is sanitized before persistence: storing a
530+// snapshot whose tombstone names a live host is accepted and reopened without
531+// Removed, instead of being rejected as durable-state corruption.
532+func TestDurableSnapshotDropsProcessLocalTombstones(t *testing.T) {
533+ o := options(t)
534+ s, err := OpenOffline(o)
535+ require.NoError(t, err)
536+ h, err := s.LoadHosts()
537+ require.NoError(t, err)
538+ require.Len(t, h.Hosts, 1)
539+ reg := h.Hosts[0].Registration
540+
541+ snap, err := s.Load()
542+ require.NoError(t, err)
543+ snap.Epoch = 5
544+ snap.Revision = 7
545+ snap.Removed = []ports.BrokerHostTombstone{{Endpoint: reg.Endpoint, Registration: reg, RetiredRevision: 6}}
546+ require.NoError(t, s.Store(snap))
547+
548+ stored, err := s.Load()
549+ require.NoError(t, err)
550+ require.Empty(t, stored.Removed)
551+ require.NoError(t, s.Close())
552+
553+ s, err = OpenOffline(o)
554+ require.NoError(t, err)
555+ defer s.Close()
556+ reopened, err := s.Load()
557+ require.NoError(t, err)
558+ require.Empty(t, reopened.Removed)
559+}
560+
561+func TestFencingAndCommitFaults(t *testing.T) {
562+ for _, point := range []string{"write", "sync", "rename", "dirsync"} {
563+ t.Run(point, func(t *testing.T) {
564+ o := options(t)
565+ s, err := OpenOffline(o)
566+ require.NoError(t, err)
567+ h, err := s.LoadHosts()
568+ require.NoError(t, err)
569+ snap, err := s.Load()
570+ require.NoError(t, err)
571+ snap.Epoch = 20
572+ snap.Revision = 1
573+ require.NoError(t, s.Store(snap))
574+ require.ErrorIs(t, s.Store(snap), ErrStale)
575+ other := snap
576+ other.Epoch = 21
577+ require.ErrorIs(t, s.Store(other), ErrStale)
578+ require.ErrorIs(t, s.ReplaceHosts(0, nil), ErrStale)
579+ changed := append([]ports.BrokerHostRecord(nil), h.Hosts...)
580+ changed[0].Policy.Trust = "new"
581+ require.ErrorIs(t, s.ReplaceHosts(h.Revision, changed), ErrStale)
582+ s.fault = func(p string) error {
583+ if p == "state.json:"+point {
584+ return errors.New("disk fault")
585+ }
586+ return nil
587+ }
588+ require.Error(t, s.ReplaceHosts(h.Revision, nil))
589+ _, err = s.LoadHosts()
590+ require.Error(t, err)
591+ require.NoError(t, s.Close())
592+ s, err = OpenOffline(o)
593+ require.NoError(t, err)
594+ defer s.Close()
595+ h, err = s.LoadHosts()
596+ require.NoError(t, err)
597+ if point == "rename" || point == "dirsync" {
598+ require.Empty(t, h.Hosts)
599+ snap.Revision++
600+ require.ErrorIs(t, s.Store(snap), ErrStale)
601+ } else {
602+ require.Len(t, h.Hosts, 1)
603+ }
604+ })
605+ }
606+ t.Run("generation and restart", func(t *testing.T) {
607+ o := options(t)
608+ s, err := OpenOffline(o)
609+ require.NoError(t, err)
610+ h, err := s.LoadHosts()
611+ require.NoError(t, err)
612+ snap, err := s.Load()
613+ require.NoError(t, err)
614+ h.Hosts[0].Registration.Generation++
615+ require.NoError(t, s.ReplaceHosts(h.Revision, h.Hosts))
616+ require.ErrorIs(t, s.Store(snap), ErrStale)
617+ require.NoError(t, s.Close())
618+ s, err = OpenOffline(o)
619+ require.NoError(t, err)
620+ defer s.Close()
621+ got, err := s.Load()
622+ require.NoError(t, err)
623+ require.Empty(t, got.Hosts)
624+ gotH, err := s.LoadHosts()
625+ require.NoError(t, err)
626+ require.Equal(t, h.Hosts, gotH.Hosts)
627+ })
628+}
A internal/adapters/brokerstore/testdata/cache-v2.json +1 -0
@@ -0,0 +1,1 @@
1+{"version": 2, "hosts": [{"target": "user@arch", "fetched_at_unix_nano": 1780000000000000000, "sessions": [{"lifecycle_id": "02000000000000000000000000000000", "name": "work", "state": "up", "ephemeral": false, "last_used_seq": 42, "active_tab_id": "t_work", "attached": false, "tab_list": [{"id": "t_work", "index": 0, "name": "shell"}]}]}]}
@@ -0,0 +1,1 @@
1+{"version": 2, "hosts": [{"target": "user@arch", "fetched_at_unix_nano": 1780000000000000000, "sessions": [{"lifecycle_id": "02000000000000000000000000000000", "name": "work", "state": "up", "ephemeral": false, "last_used_seq": 42, "active_tab_id": "t_work", "attached": false, "tab_list": [{"id": "t_work", "index": 0, "name": "shell"}]}]}]}
A internal/adapters/brokerstore/testdata/cache-v3.json +1 -0
@@ -0,0 +1,1 @@
1+{"version": 3, "hosts": [{"target": "user@arch", "fetched_at_unix_nano": 1780000000000000000, "sessions": [{"lifecycle_id": "02000000000000000000000000000000", "name": "work", "state": "up", "ephemeral": false, "last_used_seq": 42, "tabs": [{"id": "t_work", "index": 0, "name": "shell"}], "active_tab_id": "t_work", "attached": false}]}]}
@@ -0,0 +1,1 @@
1+{"version": 3, "hosts": [{"target": "user@arch", "fetched_at_unix_nano": 1780000000000000000, "sessions": [{"lifecycle_id": "02000000000000000000000000000000", "name": "work", "state": "up", "ephemeral": false, "last_used_seq": 42, "tabs": [{"id": "t_work", "index": 0, "name": "shell"}], "active_tab_id": "t_work", "attached": false}]}]}
A internal/adapters/brokerstore/testdata/cache-v4.json +1 -0
@@ -0,0 +1,1 @@
1+{"version": 4, "hosts": [{"target": "user@arch", "fetched_at_unix_nano": 1780000000000000000, "sessions": [{"lifecycle_id": "02000000000000000000000000000000", "name": "work", "state": "up", "ephemeral": false, "last_used_seq": 42, "tabs": [{"id": "t_work", "index": 0, "name": "shell"}], "active_tab_id": "t_work", "attached": false}], "incarnation": "01000000000000000000000000000000"}]}
@@ -0,0 +1,1 @@
1+{"version": 4, "hosts": [{"target": "user@arch", "fetched_at_unix_nano": 1780000000000000000, "sessions": [{"lifecycle_id": "02000000000000000000000000000000", "name": "work", "state": "up", "ephemeral": false, "last_used_seq": 42, "tabs": [{"id": "t_work", "index": 0, "name": "shell"}], "active_tab_id": "t_work", "attached": false}], "incarnation": "01000000000000000000000000000000"}]}
A internal/adapters/brokerstore/testdata/hosts-v1.json +1 -0
@@ -0,0 +1,1 @@
1+{"version": 1, "hosts": ["user@arch"]}
@@ -0,0 +1,1 @@
1+{"version": 1, "hosts": ["user@arch"]}
A internal/adapters/brokerstore/testdata/hosts-v2.json +1 -0
@@ -0,0 +1,1 @@
1+{"version": 2, "pinned": ["user@arch"], "learned": ["user@arch"]}
@@ -0,0 +1,1 @@
1+{"version": 2, "pinned": ["user@arch"], "learned": ["user@arch"]}
A internal/adapters/brokerstore/testdata/hosts-v3.json +1 -0
@@ -0,0 +1,1 @@
1+{"version": 3, "pinned": [{"endpoint": "user@arch", "incarnation": "01000000000000000000000000000000", "generation": 7}], "learned": [{"endpoint": "user@arch", "incarnation": "01000000000000000000000000000000", "generation": 7}]}
@@ -0,0 +1,1 @@
1+{"version": 3, "pinned": [{"endpoint": "user@arch", "incarnation": "01000000000000000000000000000000", "generation": 7}], "learned": [{"endpoint": "user@arch", "incarnation": "01000000000000000000000000000000", "generation": 7}]}
M internal/ports/broker.go +28 -11
@@ -69,6 +69,22 @@ const (
6969 BrokerMaxPolicyTokenBytes = 256
7070 )
7171
72+// Broker registry lifecycle sentinels. They are typed so a caller can classify
73+// why the registry refused a membership mutation without importing the use
74+// case or matching on message text.
75+var (
76+ // ErrBrokerRegistryClosed reports that the registry run has settled: its
77+ // durable writer is flushed, in-flight attempts are retired, and no further
78+ // membership mutation is acknowledged.
79+ ErrBrokerRegistryClosed = errors.New("ports: broker registry is closed")
80+ // ErrBrokerRevisionExhausted reports that the broker epoch has consumed its
81+ // entire revision series. The registry fails closed: it keeps the last valid
82+ // snapshot published and refuses further mutations instead of wrapping to
83+ // zero or publishing a revision below the newest one, either of which would
84+ // silently freeze durable persistence.
85+ ErrBrokerRevisionExhausted = errors.New("ports: broker revision series exhausted")
86+)
87+
7288 // BrokerEpoch identifies one broker process incarnation. Every broker
7389 // process samples a fresh non-zero epoch at startup; every publication
7490 // carries a revision within that epoch. Clients discard all broker state
@@ -220,7 +236,9 @@ func (t BrokerHostTombstone) Fences(candidate domain.RemoteRegistration) bool {
220236 // BrokerSnapshot is an immutable, fully defensive broker publication. Hosts
221237 // reuse the ordered RemoteHostSnapshot projection; Removed carries the
222238 // bounded tombstone set that fences stale authority. Nested slices are
223-// never mutated after publication.
239+// never mutated after publication. Removed is process-local fencing state and
240+// is never durable: a BrokerSnapshotStore persists and reloads hosts without
241+// tombstones.
224242 type BrokerSnapshot struct {
225243 Epoch BrokerEpoch
226244 Revision BrokerRevision
@@ -618,17 +636,16 @@ type BrokerSnapshotStore interface {
618636 }
619637
620638 // BrokerHostProbe observes one exact host registration without creating an
621-// attachment. The returned projection must describe the same registration,
622-// must not exceed BrokerMaxSessionsPerHost sessions (an over-bound projection
623-// is classified as an invalid response and never published), and every
624-// returned session and its catalogue values must already be fully validated
625-// and within the catalogue bounds: the registry re-checks only the session
626-// count, so malformed or over-bound session names, tab identities, states,
627-// and detail text would be published verbatim. Implementations should
639+// attachment. The returned projection must describe the same registration and
640+// must satisfy the durable projection rules (see
641+// ValidateDurableHostProjection): the registry re-checks exactly those rules,
642+// so an over-bound or catalogue-invalid session inventory is classified as an
643+// invalid response and never published. Implementations should therefore
628644 // validate the projection with catalogue.ValidateRemoteCatalog against the
629-// RemoteCatalogMax* bounds before returning it. Probe must return promptly
630-// once ctx is cancelled: the registry cancels the attempt context when a
631-// registration is replaced or removed, and when the run settles. A probe that
645+// RemoteCatalogMax* bounds before returning it rather than relying on that
646+// rejection. Probe must return promptly once ctx is cancelled: the registry
647+// cancels the attempt context when a registration is replaced or removed, and
648+// when the run settles. A probe that
632649 // ignores cancellation delays that retirement and can overlap a replacement
633650 // attempt for the same endpoint. Stale completions that arrive anyway are
634651 // fenced by epoch and registration identity before publication.
@@ -69,6 +69,22 @@ const (
69 BrokerMaxPolicyTokenBytes = 25669 BrokerMaxPolicyTokenBytes = 256
70 )70 )
71 71
72+// Broker registry lifecycle sentinels. They are typed so a caller can classify
73+// why the registry refused a membership mutation without importing the use
74+// case or matching on message text.
75+var (
76+ // ErrBrokerRegistryClosed reports that the registry run has settled: its
77+ // durable writer is flushed, in-flight attempts are retired, and no further
78+ // membership mutation is acknowledged.
79+ ErrBrokerRegistryClosed = errors.New("ports: broker registry is closed")
80+ // ErrBrokerRevisionExhausted reports that the broker epoch has consumed its
81+ // entire revision series. The registry fails closed: it keeps the last valid
82+ // snapshot published and refuses further mutations instead of wrapping to
83+ // zero or publishing a revision below the newest one, either of which would
84+ // silently freeze durable persistence.
85+ ErrBrokerRevisionExhausted = errors.New("ports: broker revision series exhausted")
86+)
87+
72 // BrokerEpoch identifies one broker process incarnation. Every broker88 // BrokerEpoch identifies one broker process incarnation. Every broker
73 // process samples a fresh non-zero epoch at startup; every publication89 // process samples a fresh non-zero epoch at startup; every publication
74 // carries a revision within that epoch. Clients discard all broker state90 // carries a revision within that epoch. Clients discard all broker state
@@ -220,7 +236,9 @@ func (t BrokerHostTombstone) Fences(candidate domain.RemoteRegistration) bool {
220 // BrokerSnapshot is an immutable, fully defensive broker publication. Hosts236 // BrokerSnapshot is an immutable, fully defensive broker publication. Hosts
221 // reuse the ordered RemoteHostSnapshot projection; Removed carries the237 // reuse the ordered RemoteHostSnapshot projection; Removed carries the
222 // bounded tombstone set that fences stale authority. Nested slices are238 // bounded tombstone set that fences stale authority. Nested slices are
223-// never mutated after publication.239+// never mutated after publication. Removed is process-local fencing state and
240+// is never durable: a BrokerSnapshotStore persists and reloads hosts without
241+// tombstones.
224 type BrokerSnapshot struct {242 type BrokerSnapshot struct {
225 Epoch BrokerEpoch243 Epoch BrokerEpoch
226 Revision BrokerRevision244 Revision BrokerRevision
@@ -618,17 +636,16 @@ type BrokerSnapshotStore interface {
618 }636 }
619 637
620 // BrokerHostProbe observes one exact host registration without creating an638 // BrokerHostProbe observes one exact host registration without creating an
621-// attachment. The returned projection must describe the same registration,639+// attachment. The returned projection must describe the same registration and
622-// must not exceed BrokerMaxSessionsPerHost sessions (an over-bound projection640+// must satisfy the durable projection rules (see
623-// is classified as an invalid response and never published), and every641+// ValidateDurableHostProjection): the registry re-checks exactly those rules,
624-// returned session and its catalogue values must already be fully validated642+// so an over-bound or catalogue-invalid session inventory is classified as an
625-// and within the catalogue bounds: the registry re-checks only the session643+// invalid response and never published. Implementations should therefore
626-// count, so malformed or over-bound session names, tab identities, states,
627-// and detail text would be published verbatim. Implementations should
628 // validate the projection with catalogue.ValidateRemoteCatalog against the644 // validate the projection with catalogue.ValidateRemoteCatalog against the
629-// RemoteCatalogMax* bounds before returning it. Probe must return promptly645+// RemoteCatalogMax* bounds before returning it rather than relying on that
630-// once ctx is cancelled: the registry cancels the attempt context when a646+// rejection. Probe must return promptly once ctx is cancelled: the registry
631-// registration is replaced or removed, and when the run settles. A probe that647+// cancels the attempt context when a registration is replaced or removed, and
648+// when the run settles. A probe that
632 // ignores cancellation delays that retirement and can overlap a replacement649 // ignores cancellation delays that retirement and can overlap a replacement
633 // attempt for the same endpoint. Stale completions that arrive anyway are650 // attempt for the same endpoint. Stale completions that arrive anyway are
634 // fenced by epoch and registration identity before publication.651 // fenced by epoch and registration identity before publication.
A internal/ports/broker_store.go +121 -0
@@ -0,0 +1,121 @@
1+package ports
2+
3+import (
4+ "errors"
5+
6+ "github.com/bnema/vev/internal/domain"
7+ "github.com/bnema/vev/internal/protocol/catalogue"
8+)
9+
10+// Durable broker store sentinels. Adapters wrap these so a caller can classify
11+// a store failure without importing the adapter: conflicting authority means
12+// refresh the token and retry, a locked store means another owner is live, and
13+// invalid state is resolved offline from the recovery record. None of them is
14+// ever retried blindly inside the store.
15+var (
16+ // ErrBrokerHostConflict reports that durable host authority changed under
17+ // the caller's expected revision, or that the supplied membership would
18+ // regress or re-trust an existing registration without a fresh identity.
19+ ErrBrokerHostConflict = errors.New("ports: broker durable host authority conflict")
20+ // ErrBrokerStoreLocked reports that another owner already holds the durable
21+ // store's lifetime lock.
22+ ErrBrokerStoreLocked = errors.New("ports: broker durable store is owned elsewhere")
23+ // ErrBrokerStoreInvalidState reports durable state that fails validation.
24+ // Such state is never silently replaced or rolled back by the store.
25+ ErrBrokerStoreInvalidState = errors.New("ports: broker durable store state is invalid")
26+)
27+
28+// BrokerHostRecord is durable registration authority. Membership and policy
29+// are never inferred from an observation or from the broker environment.
30+type BrokerHostRecord struct {
31+ Registration domain.RemoteRegistration
32+ Pinned bool
33+ Learned bool
34+ Policy BrokerPolicy
35+}
36+
37+// BrokerHosts is an optimistic-concurrency token and the ordered authority.
38+// Revision is store-local and independent of broker publication epochs.
39+type BrokerHosts struct {
40+ Revision uint64
41+ Hosts []BrokerHostRecord
42+}
43+
44+func (h BrokerHosts) Validate() error {
45+ if h.Revision == 0 {
46+ return errors.New("ports: invalid broker host state")
47+ }
48+ return ValidateBrokerHostRecords(h.Hosts)
49+}
50+
51+// ValidateBrokerHostRecords reports whether caller-supplied membership is
52+// internally valid, independently of any revision token: every registration
53+// and policy is valid, every record is anchored as pinned or learned, no
54+// endpoint repeats, and the set is within the host bound. A durable store
55+// applies this rule to caller input before its CAS, so malformed membership is
56+// a caller error rather than durable-state corruption.
57+func ValidateBrokerHostRecords(hosts []BrokerHostRecord) error {
58+ if len(hosts) > BrokerMaxHosts {
59+ return errors.New("ports: invalid broker host state")
60+ }
61+ seen := make(map[string]bool, len(hosts))
62+ for _, host := range hosts {
63+ if err := host.Registration.Validate(); err != nil {
64+ return err
65+ }
66+ if err := host.Policy.Validate(); err != nil {
67+ return err
68+ }
69+ if (!host.Pinned && !host.Learned) || seen[host.Registration.Endpoint] {
70+ return errors.New("ports: invalid broker membership")
71+ }
72+ seen[host.Registration.Endpoint] = true
73+ }
74+ return nil
75+}
76+
77+// BrokerHostStore owns durable membership separately from advisory snapshots.
78+// ReplaceHosts compares expected with the current host revision, increments it,
79+// and atomically discards observations for removed/replaced registrations or
80+// changed policies. Callers supply fresh incarnations on remove/re-add.
81+// ReplaceHosts validates the supplied membership (ValidateBrokerHostRecords)
82+// before its compare-and-swap, so malformed caller input is reported as a
83+// caller error and never as invalid durable state. Close releases exclusive
84+// ownership; no operation is valid after Close.
85+type BrokerHostStore interface {
86+ BrokerSnapshotStore
87+ LoadHosts() (BrokerHosts, error)
88+ ReplaceHosts(expected uint64, hosts []BrokerHostRecord) error
89+ Close() error
90+}
91+
92+// BrokerHostStore is a BrokerSnapshotStore: the exclusive owner of durable
93+// membership also owns the advisory snapshot, so the registry writes through
94+// one seam. This guard keeps that composition compile-checked.
95+var _ BrokerSnapshotStore = (BrokerHostStore)(nil)
96+
97+// ValidateDurableHostProjection reports whether one host projection satisfies
98+// the semantic rules of durable broker state: a published availability and
99+// failure kind inside their closed ranges, and a session inventory the
100+// catalogue accepts for the host's exact incarnation and success time. The
101+// offline store and the registry both apply this rule, so a projection the
102+// registry publishes is never rejected (and therefore silently never
103+// persisted) by the store. Transient state (Checking and the live failure
104+// cause) is deliberately not part of the durable rule.
105+func ValidateDurableHostProjection(host RemoteHostSnapshot) error {
106+ if host.Availability < domain.RemoteAvailabilityUnknown || host.Availability > domain.RemoteAvailabilityInvalidResponse {
107+ return errors.New("ports: broker host availability is out of range")
108+ }
109+ if host.LastFailure.Kind > domain.RemoteFailureInvalidResponse {
110+ return errors.New("ports: broker host failure kind is out of range")
111+ }
112+ if len(host.Sessions) == 0 {
113+ return nil
114+ }
115+ return catalogue.ValidateRemoteCatalogCacheEntries([]catalogue.RemoteCatalogCacheEntry{{
116+ Host: host.Endpoint,
117+ Incarnation: host.Registration.Incarnation,
118+ FetchedAt: host.LastSuccess,
119+ Sessions: host.Sessions,
120+ }})
121+}
@@ -0,0 +1,121 @@
1+package ports
2+
3+import (
4+ "errors"
5+
6+ "github.com/bnema/vev/internal/domain"
7+ "github.com/bnema/vev/internal/protocol/catalogue"
8+)
9+
10+// Durable broker store sentinels. Adapters wrap these so a caller can classify
11+// a store failure without importing the adapter: conflicting authority means
12+// refresh the token and retry, a locked store means another owner is live, and
13+// invalid state is resolved offline from the recovery record. None of them is
14+// ever retried blindly inside the store.
15+var (
16+ // ErrBrokerHostConflict reports that durable host authority changed under
17+ // the caller's expected revision, or that the supplied membership would
18+ // regress or re-trust an existing registration without a fresh identity.
19+ ErrBrokerHostConflict = errors.New("ports: broker durable host authority conflict")
20+ // ErrBrokerStoreLocked reports that another owner already holds the durable
21+ // store's lifetime lock.
22+ ErrBrokerStoreLocked = errors.New("ports: broker durable store is owned elsewhere")
23+ // ErrBrokerStoreInvalidState reports durable state that fails validation.
24+ // Such state is never silently replaced or rolled back by the store.
25+ ErrBrokerStoreInvalidState = errors.New("ports: broker durable store state is invalid")
26+)
27+
28+// BrokerHostRecord is durable registration authority. Membership and policy
29+// are never inferred from an observation or from the broker environment.
30+type BrokerHostRecord struct {
31+ Registration domain.RemoteRegistration
32+ Pinned bool
33+ Learned bool
34+ Policy BrokerPolicy
35+}
36+
37+// BrokerHosts is an optimistic-concurrency token and the ordered authority.
38+// Revision is store-local and independent of broker publication epochs.
39+type BrokerHosts struct {
40+ Revision uint64
41+ Hosts []BrokerHostRecord
42+}
43+
44+func (h BrokerHosts) Validate() error {
45+ if h.Revision == 0 {
46+ return errors.New("ports: invalid broker host state")
47+ }
48+ return ValidateBrokerHostRecords(h.Hosts)
49+}
50+
51+// ValidateBrokerHostRecords reports whether caller-supplied membership is
52+// internally valid, independently of any revision token: every registration
53+// and policy is valid, every record is anchored as pinned or learned, no
54+// endpoint repeats, and the set is within the host bound. A durable store
55+// applies this rule to caller input before its CAS, so malformed membership is
56+// a caller error rather than durable-state corruption.
57+func ValidateBrokerHostRecords(hosts []BrokerHostRecord) error {
58+ if len(hosts) > BrokerMaxHosts {
59+ return errors.New("ports: invalid broker host state")
60+ }
61+ seen := make(map[string]bool, len(hosts))
62+ for _, host := range hosts {
63+ if err := host.Registration.Validate(); err != nil {
64+ return err
65+ }
66+ if err := host.Policy.Validate(); err != nil {
67+ return err
68+ }
69+ if (!host.Pinned && !host.Learned) || seen[host.Registration.Endpoint] {
70+ return errors.New("ports: invalid broker membership")
71+ }
72+ seen[host.Registration.Endpoint] = true
73+ }
74+ return nil
75+}
76+
77+// BrokerHostStore owns durable membership separately from advisory snapshots.
78+// ReplaceHosts compares expected with the current host revision, increments it,
79+// and atomically discards observations for removed/replaced registrations or
80+// changed policies. Callers supply fresh incarnations on remove/re-add.
81+// ReplaceHosts validates the supplied membership (ValidateBrokerHostRecords)
82+// before its compare-and-swap, so malformed caller input is reported as a
83+// caller error and never as invalid durable state. Close releases exclusive
84+// ownership; no operation is valid after Close.
85+type BrokerHostStore interface {
86+ BrokerSnapshotStore
87+ LoadHosts() (BrokerHosts, error)
88+ ReplaceHosts(expected uint64, hosts []BrokerHostRecord) error
89+ Close() error
90+}
91+
92+// BrokerHostStore is a BrokerSnapshotStore: the exclusive owner of durable
93+// membership also owns the advisory snapshot, so the registry writes through
94+// one seam. This guard keeps that composition compile-checked.
95+var _ BrokerSnapshotStore = (BrokerHostStore)(nil)
96+
97+// ValidateDurableHostProjection reports whether one host projection satisfies
98+// the semantic rules of durable broker state: a published availability and
99+// failure kind inside their closed ranges, and a session inventory the
100+// catalogue accepts for the host's exact incarnation and success time. The
101+// offline store and the registry both apply this rule, so a projection the
102+// registry publishes is never rejected (and therefore silently never
103+// persisted) by the store. Transient state (Checking and the live failure
104+// cause) is deliberately not part of the durable rule.
105+func ValidateDurableHostProjection(host RemoteHostSnapshot) error {
106+ if host.Availability < domain.RemoteAvailabilityUnknown || host.Availability > domain.RemoteAvailabilityInvalidResponse {
107+ return errors.New("ports: broker host availability is out of range")
108+ }
109+ if host.LastFailure.Kind > domain.RemoteFailureInvalidResponse {
110+ return errors.New("ports: broker host failure kind is out of range")
111+ }
112+ if len(host.Sessions) == 0 {
113+ return nil
114+ }
115+ return catalogue.ValidateRemoteCatalogCacheEntries([]catalogue.RemoteCatalogCacheEntry{{
116+ Host: host.Endpoint,
117+ Incarnation: host.Registration.Incarnation,
118+ FetchedAt: host.LastSuccess,
119+ Sessions: host.Sessions,
120+ }})
121+}
M internal/ports/broker_test.go +59 -0
@@ -4,6 +4,7 @@ import (
44 "reflect"
55 "strings"
66 "testing"
7+ "time"
78
89 "github.com/bnema/vev/internal/domain"
910 "github.com/bnema/vev/internal/protocol"
@@ -53,6 +54,64 @@ func validBrokerSnapshot() BrokerSnapshot {
5354 }
5455 }
5556
57+// TestBrokerHostStoreIsASnapshotStore pins the interface guard: the exclusive
58+// owner of durable membership also owns the advisory snapshot, so a
59+// BrokerHostStore must stay usable wherever a BrokerSnapshotStore is required.
60+func TestBrokerHostStoreIsASnapshotStore(t *testing.T) {
61+ host := reflect.TypeOf((*BrokerHostStore)(nil)).Elem()
62+ snapshot := reflect.TypeOf((*BrokerSnapshotStore)(nil)).Elem()
63+ if !host.Implements(snapshot) {
64+ t.Fatal("BrokerHostStore must implement BrokerSnapshotStore")
65+ }
66+}
67+
68+// TestValidateDurableHostProjection matches the durable rules the offline store
69+// enforces and the registry re-checks before adopting an observation.
70+func TestValidateDurableHostProjection(t *testing.T) {
71+ session := catalogue.RemoteCatalogSession{
72+ LifecycleID: testBrokerLifecycle(),
73+ Name: "work",
74+ State: catalogue.RemoteCatalogSessionUp,
75+ Tabs: []catalogue.RemoteCatalogTab{},
76+ }
77+ tests := []struct {
78+ name string
79+ mutate func(*RemoteHostSnapshot)
80+ wantErr bool
81+ }{
82+ {name: "empty inventory", mutate: func(*RemoteHostSnapshot) {}},
83+ {name: "valid inventory", mutate: func(h *RemoteHostSnapshot) { h.Sessions = []catalogue.RemoteCatalogSession{session} }},
84+ {name: "zero availability", mutate: func(h *RemoteHostSnapshot) { h.Availability = 0 }, wantErr: true},
85+ {name: "unknown availability", mutate: func(h *RemoteHostSnapshot) { h.Availability = domain.RemoteAvailabilityUnknown }},
86+ {name: "availability past the closed range", mutate: func(h *RemoteHostSnapshot) { h.Availability = domain.RemoteAvailabilityInvalidResponse + 1 }, wantErr: true},
87+ {name: "failure kind past the closed range", mutate: func(h *RemoteHostSnapshot) { h.LastFailure.Kind = domain.RemoteFailureInvalidResponse + 1 }, wantErr: true},
88+ {name: "missing lifecycle identity", mutate: func(h *RemoteHostSnapshot) {
89+ h.Sessions = []catalogue.RemoteCatalogSession{{Name: "work", State: catalogue.RemoteCatalogSessionUp, Tabs: []catalogue.RemoteCatalogTab{}}}
90+ }, wantErr: true},
91+ {name: "absent tab list", mutate: func(h *RemoteHostSnapshot) {
92+ broken := session
93+ broken.Tabs = nil
94+ h.Sessions = []catalogue.RemoteCatalogSession{broken}
95+ }, wantErr: true},
96+ {name: "sessions past the catalogue bound", mutate: func(h *RemoteHostSnapshot) {
97+ h.Sessions = make([]catalogue.RemoteCatalogSession, catalogue.RemoteCatalogMaxSessions+1)
98+ for i := range h.Sessions {
99+ h.Sessions[i] = session
100+ }
101+ }, wantErr: true},
102+ }
103+ for _, tc := range tests {
104+ t.Run(tc.name, func(t *testing.T) {
105+ host := testBrokerHost("user@arch")
106+ host.LastSuccess = time.Unix(50, 0)
107+ tc.mutate(&host)
108+ if gotErr := ValidateDurableHostProjection(host); (gotErr != nil) != tc.wantErr {
109+ t.Fatalf("ValidateDurableHostProjection() = %v, wantErr %t", gotErr, tc.wantErr)
110+ }
111+ })
112+ }
113+}
114+
56115 func TestBrokerSnapshotValidateBounds(t *testing.T) {
57116 tests := []struct {
58117 name string
@@ -4,6 +4,7 @@ import (
4 "reflect"4 "reflect"
5 "strings"5 "strings"
6 "testing"6 "testing"
7+ "time"
7 8
8 "github.com/bnema/vev/internal/domain"9 "github.com/bnema/vev/internal/domain"
9 "github.com/bnema/vev/internal/protocol"10 "github.com/bnema/vev/internal/protocol"
@@ -53,6 +54,64 @@ func validBrokerSnapshot() BrokerSnapshot {
53 }54 }
54 }55 }
55 56
57+// TestBrokerHostStoreIsASnapshotStore pins the interface guard: the exclusive
58+// owner of durable membership also owns the advisory snapshot, so a
59+// BrokerHostStore must stay usable wherever a BrokerSnapshotStore is required.
60+func TestBrokerHostStoreIsASnapshotStore(t *testing.T) {
61+ host := reflect.TypeOf((*BrokerHostStore)(nil)).Elem()
62+ snapshot := reflect.TypeOf((*BrokerSnapshotStore)(nil)).Elem()
63+ if !host.Implements(snapshot) {
64+ t.Fatal("BrokerHostStore must implement BrokerSnapshotStore")
65+ }
66+}
67+
68+// TestValidateDurableHostProjection matches the durable rules the offline store
69+// enforces and the registry re-checks before adopting an observation.
70+func TestValidateDurableHostProjection(t *testing.T) {
71+ session := catalogue.RemoteCatalogSession{
72+ LifecycleID: testBrokerLifecycle(),
73+ Name: "work",
74+ State: catalogue.RemoteCatalogSessionUp,
75+ Tabs: []catalogue.RemoteCatalogTab{},
76+ }
77+ tests := []struct {
78+ name string
79+ mutate func(*RemoteHostSnapshot)
80+ wantErr bool
81+ }{
82+ {name: "empty inventory", mutate: func(*RemoteHostSnapshot) {}},
83+ {name: "valid inventory", mutate: func(h *RemoteHostSnapshot) { h.Sessions = []catalogue.RemoteCatalogSession{session} }},
84+ {name: "zero availability", mutate: func(h *RemoteHostSnapshot) { h.Availability = 0 }, wantErr: true},
85+ {name: "unknown availability", mutate: func(h *RemoteHostSnapshot) { h.Availability = domain.RemoteAvailabilityUnknown }},
86+ {name: "availability past the closed range", mutate: func(h *RemoteHostSnapshot) { h.Availability = domain.RemoteAvailabilityInvalidResponse + 1 }, wantErr: true},
87+ {name: "failure kind past the closed range", mutate: func(h *RemoteHostSnapshot) { h.LastFailure.Kind = domain.RemoteFailureInvalidResponse + 1 }, wantErr: true},
88+ {name: "missing lifecycle identity", mutate: func(h *RemoteHostSnapshot) {
89+ h.Sessions = []catalogue.RemoteCatalogSession{{Name: "work", State: catalogue.RemoteCatalogSessionUp, Tabs: []catalogue.RemoteCatalogTab{}}}
90+ }, wantErr: true},
91+ {name: "absent tab list", mutate: func(h *RemoteHostSnapshot) {
92+ broken := session
93+ broken.Tabs = nil
94+ h.Sessions = []catalogue.RemoteCatalogSession{broken}
95+ }, wantErr: true},
96+ {name: "sessions past the catalogue bound", mutate: func(h *RemoteHostSnapshot) {
97+ h.Sessions = make([]catalogue.RemoteCatalogSession, catalogue.RemoteCatalogMaxSessions+1)
98+ for i := range h.Sessions {
99+ h.Sessions[i] = session
100+ }
101+ }, wantErr: true},
102+ }
103+ for _, tc := range tests {
104+ t.Run(tc.name, func(t *testing.T) {
105+ host := testBrokerHost("user@arch")
106+ host.LastSuccess = time.Unix(50, 0)
107+ tc.mutate(&host)
108+ if gotErr := ValidateDurableHostProjection(host); (gotErr != nil) != tc.wantErr {
109+ t.Fatalf("ValidateDurableHostProjection() = %v, wantErr %t", gotErr, tc.wantErr)
110+ }
111+ })
112+ }
113+}
114+
56 func TestBrokerSnapshotValidateBounds(t *testing.T) {115 func TestBrokerSnapshotValidateBounds(t *testing.T) {
57 tests := []struct {116 tests := []struct {
58 name string117 name string
A internal/ports/mocks/broker_store_mock.go +295 -0
@@ -0,0 +1,295 @@
1+// Code generated by mockery; DO NOT EDIT.
2+// github.com/vektra/mockery
3+// template: testify
4+
5+package portsmocks
6+
7+import (
8+ "github.com/bnema/vev/internal/ports"
9+ mock "github.com/stretchr/testify/mock"
10+)
11+
12+// NewMockBrokerHostStore creates a new instance of MockBrokerHostStore. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
13+// The first argument is typically a *testing.T value.
14+func NewMockBrokerHostStore(t interface {
15+ mock.TestingT
16+ Cleanup(func())
17+}) *MockBrokerHostStore {
18+ mock := &MockBrokerHostStore{}
19+ mock.Mock.Test(t)
20+
21+ t.Cleanup(func() { mock.AssertExpectations(t) })
22+
23+ return mock
24+}
25+
26+// MockBrokerHostStore is an autogenerated mock type for the BrokerHostStore type
27+type MockBrokerHostStore struct {
28+ mock.Mock
29+}
30+
31+type MockBrokerHostStore_Expecter struct {
32+ mock *mock.Mock
33+}
34+
35+func (_m *MockBrokerHostStore) EXPECT() *MockBrokerHostStore_Expecter {
36+ return &MockBrokerHostStore_Expecter{mock: &_m.Mock}
37+}
38+
39+// Close provides a mock function for the type MockBrokerHostStore
40+func (_mock *MockBrokerHostStore) Close() error {
41+ ret := _mock.Called()
42+
43+ if len(ret) == 0 {
44+ panic("no return value specified for Close")
45+ }
46+
47+ var r0 error
48+ if returnFunc, ok := ret.Get(0).(func() error); ok {
49+ r0 = returnFunc()
50+ } else {
51+ r0 = ret.Error(0)
52+ }
53+ return r0
54+}
55+
56+// MockBrokerHostStore_Close_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Close'
57+type MockBrokerHostStore_Close_Call struct {
58+ *mock.Call
59+}
60+
61+// Close is a helper method to define mock.On call
62+func (_e *MockBrokerHostStore_Expecter) Close() *MockBrokerHostStore_Close_Call {
63+ return &MockBrokerHostStore_Close_Call{Call: _e.mock.On("Close")}
64+}
65+
66+func (_c *MockBrokerHostStore_Close_Call) Run(run func()) *MockBrokerHostStore_Close_Call {
67+ _c.Call.Run(func(args mock.Arguments) {
68+ run()
69+ })
70+ return _c
71+}
72+
73+func (_c *MockBrokerHostStore_Close_Call) Return(err error) *MockBrokerHostStore_Close_Call {
74+ _c.Call.Return(err)
75+ return _c
76+}
77+
78+func (_c *MockBrokerHostStore_Close_Call) RunAndReturn(run func() error) *MockBrokerHostStore_Close_Call {
79+ _c.Call.Return(run)
80+ return _c
81+}
82+
83+// Load provides a mock function for the type MockBrokerHostStore
84+func (_mock *MockBrokerHostStore) Load() (ports.BrokerSnapshot, error) {
85+ ret := _mock.Called()
86+
87+ if len(ret) == 0 {
88+ panic("no return value specified for Load")
89+ }
90+
91+ var r0 ports.BrokerSnapshot
92+ var r1 error
93+ if returnFunc, ok := ret.Get(0).(func() (ports.BrokerSnapshot, error)); ok {
94+ return returnFunc()
95+ }
96+ if returnFunc, ok := ret.Get(0).(func() ports.BrokerSnapshot); ok {
97+ r0 = returnFunc()
98+ } else {
99+ r0 = ret.Get(0).(ports.BrokerSnapshot)
100+ }
101+ if returnFunc, ok := ret.Get(1).(func() error); ok {
102+ r1 = returnFunc()
103+ } else {
104+ r1 = ret.Error(1)
105+ }
106+ return r0, r1
107+}
108+
109+// MockBrokerHostStore_Load_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Load'
110+type MockBrokerHostStore_Load_Call struct {
111+ *mock.Call
112+}
113+
114+// Load is a helper method to define mock.On call
115+func (_e *MockBrokerHostStore_Expecter) Load() *MockBrokerHostStore_Load_Call {
116+ return &MockBrokerHostStore_Load_Call{Call: _e.mock.On("Load")}
117+}
118+
119+func (_c *MockBrokerHostStore_Load_Call) Run(run func()) *MockBrokerHostStore_Load_Call {
120+ _c.Call.Run(func(args mock.Arguments) {
121+ run()
122+ })
123+ return _c
124+}
125+
126+func (_c *MockBrokerHostStore_Load_Call) Return(brokerSnapshot ports.BrokerSnapshot, err error) *MockBrokerHostStore_Load_Call {
127+ _c.Call.Return(brokerSnapshot, err)
128+ return _c
129+}
130+
131+func (_c *MockBrokerHostStore_Load_Call) RunAndReturn(run func() (ports.BrokerSnapshot, error)) *MockBrokerHostStore_Load_Call {
132+ _c.Call.Return(run)
133+ return _c
134+}
135+
136+// LoadHosts provides a mock function for the type MockBrokerHostStore
137+func (_mock *MockBrokerHostStore) LoadHosts() (ports.BrokerHosts, error) {
138+ ret := _mock.Called()
139+
140+ if len(ret) == 0 {
141+ panic("no return value specified for LoadHosts")
142+ }
143+
144+ var r0 ports.BrokerHosts
145+ var r1 error
146+ if returnFunc, ok := ret.Get(0).(func() (ports.BrokerHosts, error)); ok {
147+ return returnFunc()
148+ }
149+ if returnFunc, ok := ret.Get(0).(func() ports.BrokerHosts); ok {
150+ r0 = returnFunc()
151+ } else {
152+ r0 = ret.Get(0).(ports.BrokerHosts)
153+ }
154+ if returnFunc, ok := ret.Get(1).(func() error); ok {
155+ r1 = returnFunc()
156+ } else {
157+ r1 = ret.Error(1)
158+ }
159+ return r0, r1
160+}
161+
162+// MockBrokerHostStore_LoadHosts_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'LoadHosts'
163+type MockBrokerHostStore_LoadHosts_Call struct {
164+ *mock.Call
165+}
166+
167+// LoadHosts is a helper method to define mock.On call
168+func (_e *MockBrokerHostStore_Expecter) LoadHosts() *MockBrokerHostStore_LoadHosts_Call {
169+ return &MockBrokerHostStore_LoadHosts_Call{Call: _e.mock.On("LoadHosts")}
170+}
171+
172+func (_c *MockBrokerHostStore_LoadHosts_Call) Run(run func()) *MockBrokerHostStore_LoadHosts_Call {
173+ _c.Call.Run(func(args mock.Arguments) {
174+ run()
175+ })
176+ return _c
177+}
178+
179+func (_c *MockBrokerHostStore_LoadHosts_Call) Return(brokerHosts ports.BrokerHosts, err error) *MockBrokerHostStore_LoadHosts_Call {
180+ _c.Call.Return(brokerHosts, err)
181+ return _c
182+}
183+
184+func (_c *MockBrokerHostStore_LoadHosts_Call) RunAndReturn(run func() (ports.BrokerHosts, error)) *MockBrokerHostStore_LoadHosts_Call {
185+ _c.Call.Return(run)
186+ return _c
187+}
188+
189+// ReplaceHosts provides a mock function for the type MockBrokerHostStore
190+func (_mock *MockBrokerHostStore) ReplaceHosts(expected uint64, hosts []ports.BrokerHostRecord) error {
191+ ret := _mock.Called(expected, hosts)
192+
193+ if len(ret) == 0 {
194+ panic("no return value specified for ReplaceHosts")
195+ }
196+
197+ var r0 error
198+ if returnFunc, ok := ret.Get(0).(func(uint64, []ports.BrokerHostRecord) error); ok {
199+ r0 = returnFunc(expected, hosts)
200+ } else {
201+ r0 = ret.Error(0)
202+ }
203+ return r0
204+}
205+
206+// MockBrokerHostStore_ReplaceHosts_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'ReplaceHosts'
207+type MockBrokerHostStore_ReplaceHosts_Call struct {
208+ *mock.Call
209+}
210+
211+// ReplaceHosts is a helper method to define mock.On call
212+// - expected uint64
213+// - hosts []ports.BrokerHostRecord
214+func (_e *MockBrokerHostStore_Expecter) ReplaceHosts(expected any, hosts any) *MockBrokerHostStore_ReplaceHosts_Call {
215+ return &MockBrokerHostStore_ReplaceHosts_Call{Call: _e.mock.On("ReplaceHosts", expected, hosts)}
216+}
217+
218+func (_c *MockBrokerHostStore_ReplaceHosts_Call) Run(run func(expected uint64, hosts []ports.BrokerHostRecord)) *MockBrokerHostStore_ReplaceHosts_Call {
219+ _c.Call.Run(func(args mock.Arguments) {
220+ var arg0 uint64
221+ if args[0] != nil {
222+ arg0 = args[0].(uint64)
223+ }
224+ var arg1 []ports.BrokerHostRecord
225+ if args[1] != nil {
226+ arg1 = args[1].([]ports.BrokerHostRecord)
227+ }
228+ run(
229+ arg0,
230+ arg1,
231+ )
232+ })
233+ return _c
234+}
235+
236+func (_c *MockBrokerHostStore_ReplaceHosts_Call) Return(err error) *MockBrokerHostStore_ReplaceHosts_Call {
237+ _c.Call.Return(err)
238+ return _c
239+}
240+
241+func (_c *MockBrokerHostStore_ReplaceHosts_Call) RunAndReturn(run func(expected uint64, hosts []ports.BrokerHostRecord) error) *MockBrokerHostStore_ReplaceHosts_Call {
242+ _c.Call.Return(run)
243+ return _c
244+}
245+
246+// Store provides a mock function for the type MockBrokerHostStore
247+func (_mock *MockBrokerHostStore) Store(brokerSnapshot ports.BrokerSnapshot) error {
248+ ret := _mock.Called(brokerSnapshot)
249+
250+ if len(ret) == 0 {
251+ panic("no return value specified for Store")
252+ }
253+
254+ var r0 error
255+ if returnFunc, ok := ret.Get(0).(func(ports.BrokerSnapshot) error); ok {
256+ r0 = returnFunc(brokerSnapshot)
257+ } else {
258+ r0 = ret.Error(0)
259+ }
260+ return r0
261+}
262+
263+// MockBrokerHostStore_Store_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Store'
264+type MockBrokerHostStore_Store_Call struct {
265+ *mock.Call
266+}
267+
268+// Store is a helper method to define mock.On call
269+// - brokerSnapshot ports.BrokerSnapshot
270+func (_e *MockBrokerHostStore_Expecter) Store(brokerSnapshot any) *MockBrokerHostStore_Store_Call {
271+ return &MockBrokerHostStore_Store_Call{Call: _e.mock.On("Store", brokerSnapshot)}
272+}
273+
274+func (_c *MockBrokerHostStore_Store_Call) Run(run func(brokerSnapshot ports.BrokerSnapshot)) *MockBrokerHostStore_Store_Call {
275+ _c.Call.Run(func(args mock.Arguments) {
276+ var arg0 ports.BrokerSnapshot
277+ if args[0] != nil {
278+ arg0 = args[0].(ports.BrokerSnapshot)
279+ }
280+ run(
281+ arg0,
282+ )
283+ })
284+ return _c
285+}
286+
287+func (_c *MockBrokerHostStore_Store_Call) Return(err error) *MockBrokerHostStore_Store_Call {
288+ _c.Call.Return(err)
289+ return _c
290+}
291+
292+func (_c *MockBrokerHostStore_Store_Call) RunAndReturn(run func(brokerSnapshot ports.BrokerSnapshot) error) *MockBrokerHostStore_Store_Call {
293+ _c.Call.Return(run)
294+ return _c
295+}
@@ -0,0 +1,295 @@
1+// Code generated by mockery; DO NOT EDIT.
2+// github.com/vektra/mockery
3+// template: testify
4+
5+package portsmocks
6+
7+import (
8+ "github.com/bnema/vev/internal/ports"
9+ mock "github.com/stretchr/testify/mock"
10+)
11+
12+// NewMockBrokerHostStore creates a new instance of MockBrokerHostStore. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
13+// The first argument is typically a *testing.T value.
14+func NewMockBrokerHostStore(t interface {
15+ mock.TestingT
16+ Cleanup(func())
17+}) *MockBrokerHostStore {
18+ mock := &MockBrokerHostStore{}
19+ mock.Mock.Test(t)
20+
21+ t.Cleanup(func() { mock.AssertExpectations(t) })
22+
23+ return mock
24+}
25+
26+// MockBrokerHostStore is an autogenerated mock type for the BrokerHostStore type
27+type MockBrokerHostStore struct {
28+ mock.Mock
29+}
30+
31+type MockBrokerHostStore_Expecter struct {
32+ mock *mock.Mock
33+}
34+
35+func (_m *MockBrokerHostStore) EXPECT() *MockBrokerHostStore_Expecter {
36+ return &MockBrokerHostStore_Expecter{mock: &_m.Mock}
37+}
38+
39+// Close provides a mock function for the type MockBrokerHostStore
40+func (_mock *MockBrokerHostStore) Close() error {
41+ ret := _mock.Called()
42+
43+ if len(ret) == 0 {
44+ panic("no return value specified for Close")
45+ }
46+
47+ var r0 error
48+ if returnFunc, ok := ret.Get(0).(func() error); ok {
49+ r0 = returnFunc()
50+ } else {
51+ r0 = ret.Error(0)
52+ }
53+ return r0
54+}
55+
56+// MockBrokerHostStore_Close_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Close'
57+type MockBrokerHostStore_Close_Call struct {
58+ *mock.Call
59+}
60+
61+// Close is a helper method to define mock.On call
62+func (_e *MockBrokerHostStore_Expecter) Close() *MockBrokerHostStore_Close_Call {
63+ return &MockBrokerHostStore_Close_Call{Call: _e.mock.On("Close")}
64+}
65+
66+func (_c *MockBrokerHostStore_Close_Call) Run(run func()) *MockBrokerHostStore_Close_Call {
67+ _c.Call.Run(func(args mock.Arguments) {
68+ run()
69+ })
70+ return _c
71+}
72+
73+func (_c *MockBrokerHostStore_Close_Call) Return(err error) *MockBrokerHostStore_Close_Call {
74+ _c.Call.Return(err)
75+ return _c
76+}
77+
78+func (_c *MockBrokerHostStore_Close_Call) RunAndReturn(run func() error) *MockBrokerHostStore_Close_Call {
79+ _c.Call.Return(run)
80+ return _c
81+}
82+
83+// Load provides a mock function for the type MockBrokerHostStore
84+func (_mock *MockBrokerHostStore) Load() (ports.BrokerSnapshot, error) {
85+ ret := _mock.Called()
86+
87+ if len(ret) == 0 {
88+ panic("no return value specified for Load")
89+ }
90+
91+ var r0 ports.BrokerSnapshot
92+ var r1 error
93+ if returnFunc, ok := ret.Get(0).(func() (ports.BrokerSnapshot, error)); ok {
94+ return returnFunc()
95+ }
96+ if returnFunc, ok := ret.Get(0).(func() ports.BrokerSnapshot); ok {
97+ r0 = returnFunc()
98+ } else {
99+ r0 = ret.Get(0).(ports.BrokerSnapshot)
100+ }
101+ if returnFunc, ok := ret.Get(1).(func() error); ok {
102+ r1 = returnFunc()
103+ } else {
104+ r1 = ret.Error(1)
105+ }
106+ return r0, r1
107+}
108+
109+// MockBrokerHostStore_Load_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Load'
110+type MockBrokerHostStore_Load_Call struct {
111+ *mock.Call
112+}
113+
114+// Load is a helper method to define mock.On call
115+func (_e *MockBrokerHostStore_Expecter) Load() *MockBrokerHostStore_Load_Call {
116+ return &MockBrokerHostStore_Load_Call{Call: _e.mock.On("Load")}
117+}
118+
119+func (_c *MockBrokerHostStore_Load_Call) Run(run func()) *MockBrokerHostStore_Load_Call {
120+ _c.Call.Run(func(args mock.Arguments) {
121+ run()
122+ })
123+ return _c
124+}
125+
126+func (_c *MockBrokerHostStore_Load_Call) Return(brokerSnapshot ports.BrokerSnapshot, err error) *MockBrokerHostStore_Load_Call {
127+ _c.Call.Return(brokerSnapshot, err)
128+ return _c
129+}
130+
131+func (_c *MockBrokerHostStore_Load_Call) RunAndReturn(run func() (ports.BrokerSnapshot, error)) *MockBrokerHostStore_Load_Call {
132+ _c.Call.Return(run)
133+ return _c
134+}
135+
136+// LoadHosts provides a mock function for the type MockBrokerHostStore
137+func (_mock *MockBrokerHostStore) LoadHosts() (ports.BrokerHosts, error) {
138+ ret := _mock.Called()
139+
140+ if len(ret) == 0 {
141+ panic("no return value specified for LoadHosts")
142+ }
143+
144+ var r0 ports.BrokerHosts
145+ var r1 error
146+ if returnFunc, ok := ret.Get(0).(func() (ports.BrokerHosts, error)); ok {
147+ return returnFunc()
148+ }
149+ if returnFunc, ok := ret.Get(0).(func() ports.BrokerHosts); ok {
150+ r0 = returnFunc()
151+ } else {
152+ r0 = ret.Get(0).(ports.BrokerHosts)
153+ }
154+ if returnFunc, ok := ret.Get(1).(func() error); ok {
155+ r1 = returnFunc()
156+ } else {
157+ r1 = ret.Error(1)
158+ }
159+ return r0, r1
160+}
161+
162+// MockBrokerHostStore_LoadHosts_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'LoadHosts'
163+type MockBrokerHostStore_LoadHosts_Call struct {
164+ *mock.Call
165+}
166+
167+// LoadHosts is a helper method to define mock.On call
168+func (_e *MockBrokerHostStore_Expecter) LoadHosts() *MockBrokerHostStore_LoadHosts_Call {
169+ return &MockBrokerHostStore_LoadHosts_Call{Call: _e.mock.On("LoadHosts")}
170+}
171+
172+func (_c *MockBrokerHostStore_LoadHosts_Call) Run(run func()) *MockBrokerHostStore_LoadHosts_Call {
173+ _c.Call.Run(func(args mock.Arguments) {
174+ run()
175+ })
176+ return _c
177+}
178+
179+func (_c *MockBrokerHostStore_LoadHosts_Call) Return(brokerHosts ports.BrokerHosts, err error) *MockBrokerHostStore_LoadHosts_Call {
180+ _c.Call.Return(brokerHosts, err)
181+ return _c
182+}
183+
184+func (_c *MockBrokerHostStore_LoadHosts_Call) RunAndReturn(run func() (ports.BrokerHosts, error)) *MockBrokerHostStore_LoadHosts_Call {
185+ _c.Call.Return(run)
186+ return _c
187+}
188+
189+// ReplaceHosts provides a mock function for the type MockBrokerHostStore
190+func (_mock *MockBrokerHostStore) ReplaceHosts(expected uint64, hosts []ports.BrokerHostRecord) error {
191+ ret := _mock.Called(expected, hosts)
192+
193+ if len(ret) == 0 {
194+ panic("no return value specified for ReplaceHosts")
195+ }
196+
197+ var r0 error
198+ if returnFunc, ok := ret.Get(0).(func(uint64, []ports.BrokerHostRecord) error); ok {
199+ r0 = returnFunc(expected, hosts)
200+ } else {
201+ r0 = ret.Error(0)
202+ }
203+ return r0
204+}
205+
206+// MockBrokerHostStore_ReplaceHosts_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'ReplaceHosts'
207+type MockBrokerHostStore_ReplaceHosts_Call struct {
208+ *mock.Call
209+}
210+
211+// ReplaceHosts is a helper method to define mock.On call
212+// - expected uint64
213+// - hosts []ports.BrokerHostRecord
214+func (_e *MockBrokerHostStore_Expecter) ReplaceHosts(expected any, hosts any) *MockBrokerHostStore_ReplaceHosts_Call {
215+ return &MockBrokerHostStore_ReplaceHosts_Call{Call: _e.mock.On("ReplaceHosts", expected, hosts)}
216+}
217+
218+func (_c *MockBrokerHostStore_ReplaceHosts_Call) Run(run func(expected uint64, hosts []ports.BrokerHostRecord)) *MockBrokerHostStore_ReplaceHosts_Call {
219+ _c.Call.Run(func(args mock.Arguments) {
220+ var arg0 uint64
221+ if args[0] != nil {
222+ arg0 = args[0].(uint64)
223+ }
224+ var arg1 []ports.BrokerHostRecord
225+ if args[1] != nil {
226+ arg1 = args[1].([]ports.BrokerHostRecord)
227+ }
228+ run(
229+ arg0,
230+ arg1,
231+ )
232+ })
233+ return _c
234+}
235+
236+func (_c *MockBrokerHostStore_ReplaceHosts_Call) Return(err error) *MockBrokerHostStore_ReplaceHosts_Call {
237+ _c.Call.Return(err)
238+ return _c
239+}
240+
241+func (_c *MockBrokerHostStore_ReplaceHosts_Call) RunAndReturn(run func(expected uint64, hosts []ports.BrokerHostRecord) error) *MockBrokerHostStore_ReplaceHosts_Call {
242+ _c.Call.Return(run)
243+ return _c
244+}
245+
246+// Store provides a mock function for the type MockBrokerHostStore
247+func (_mock *MockBrokerHostStore) Store(brokerSnapshot ports.BrokerSnapshot) error {
248+ ret := _mock.Called(brokerSnapshot)
249+
250+ if len(ret) == 0 {
251+ panic("no return value specified for Store")
252+ }
253+
254+ var r0 error
255+ if returnFunc, ok := ret.Get(0).(func(ports.BrokerSnapshot) error); ok {
256+ r0 = returnFunc(brokerSnapshot)
257+ } else {
258+ r0 = ret.Error(0)
259+ }
260+ return r0
261+}
262+
263+// MockBrokerHostStore_Store_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Store'
264+type MockBrokerHostStore_Store_Call struct {
265+ *mock.Call
266+}
267+
268+// Store is a helper method to define mock.On call
269+// - brokerSnapshot ports.BrokerSnapshot
270+func (_e *MockBrokerHostStore_Expecter) Store(brokerSnapshot any) *MockBrokerHostStore_Store_Call {
271+ return &MockBrokerHostStore_Store_Call{Call: _e.mock.On("Store", brokerSnapshot)}
272+}
273+
274+func (_c *MockBrokerHostStore_Store_Call) Run(run func(brokerSnapshot ports.BrokerSnapshot)) *MockBrokerHostStore_Store_Call {
275+ _c.Call.Run(func(args mock.Arguments) {
276+ var arg0 ports.BrokerSnapshot
277+ if args[0] != nil {
278+ arg0 = args[0].(ports.BrokerSnapshot)
279+ }
280+ run(
281+ arg0,
282+ )
283+ })
284+ return _c
285+}
286+
287+func (_c *MockBrokerHostStore_Store_Call) Return(err error) *MockBrokerHostStore_Store_Call {
288+ _c.Call.Return(err)
289+ return _c
290+}
291+
292+func (_c *MockBrokerHostStore_Store_Call) RunAndReturn(run func(brokerSnapshot ports.BrokerSnapshot) error) *MockBrokerHostStore_Store_Call {
293+ _c.Call.Return(run)
294+ return _c
295+}
M internal/ports/remote_state.go +7 -2
@@ -28,12 +28,17 @@ type RemoteHostSnapshot struct {
2828 Sessions []catalogue.RemoteCatalogSession
2929 }
3030
31-// Clone returns a defensive copy with an independent session slice.
31+// Clone returns a defensive copy with an independent session slice. Each
32+// session keeps the exact nil-ness of its tab list: an empty but present list
33+// is catalogue-valid and must not become absent through a copy.
3234 func (s RemoteHostSnapshot) Clone() RemoteHostSnapshot {
3335 out := s
3436 out.Sessions = append([]catalogue.RemoteCatalogSession(nil), s.Sessions...)
3537 for i := range out.Sessions {
36- out.Sessions[i].Tabs = append([]catalogue.RemoteCatalogTab(nil), out.Sessions[i].Tabs...)
38+ if s.Sessions[i].Tabs == nil {
39+ continue
40+ }
41+ out.Sessions[i].Tabs = append(make([]catalogue.RemoteCatalogTab, 0, len(s.Sessions[i].Tabs)), s.Sessions[i].Tabs...)
3742 }
3843 return out
3944 }
@@ -28,12 +28,17 @@ type RemoteHostSnapshot struct {
28 Sessions []catalogue.RemoteCatalogSession28 Sessions []catalogue.RemoteCatalogSession
29 }29 }
30 30
31-// Clone returns a defensive copy with an independent session slice.31+// Clone returns a defensive copy with an independent session slice. Each
32+// session keeps the exact nil-ness of its tab list: an empty but present list
33+// is catalogue-valid and must not become absent through a copy.
32 func (s RemoteHostSnapshot) Clone() RemoteHostSnapshot {34 func (s RemoteHostSnapshot) Clone() RemoteHostSnapshot {
33 out := s35 out := s
34 out.Sessions = append([]catalogue.RemoteCatalogSession(nil), s.Sessions...)36 out.Sessions = append([]catalogue.RemoteCatalogSession(nil), s.Sessions...)
35 for i := range out.Sessions {37 for i := range out.Sessions {
36- out.Sessions[i].Tabs = append([]catalogue.RemoteCatalogTab(nil), out.Sessions[i].Tabs...)38+ if s.Sessions[i].Tabs == nil {
39+ continue
40+ }
41+ out.Sessions[i].Tabs = append(make([]catalogue.RemoteCatalogTab, 0, len(s.Sessions[i].Tabs)), s.Sessions[i].Tabs...)
37 }42 }
38 return out43 return out
39 }44 }
M internal/ports/remote_state_test.go +8 -1
@@ -72,7 +72,7 @@ func TestRemoteDirectorySnapshotCloneIsDefensive(t *testing.T) {
7272 func TestRemoteHostSnapshotCloneIsDefensive(t *testing.T) {
7373 original := RemoteHostSnapshot{
7474 Endpoint: "user@arch",
75- Sessions: []catalogue.RemoteCatalogSession{{Name: "work"}},
75+ Sessions: []catalogue.RemoteCatalogSession{{Name: "work"}, {Name: "empty", Tabs: []catalogue.RemoteCatalogTab{}}},
7676 }
7777 cloned := original.Clone()
7878 cloned.Sessions[0].Name = "mutated"
@@ -82,4 +82,11 @@ func TestRemoteHostSnapshotCloneIsDefensive(t *testing.T) {
8282 if (RemoteHostSnapshot{}).Clone().Sessions != nil {
8383 t.Fatal("Clone() of a session-less host must stay nil")
8484 }
85+ if cloned.Sessions[1].Tabs == nil {
86+ t.Fatal("Clone() must keep an empty but present tab list")
87+ }
88+ cloned.Sessions[1].Tabs = append(cloned.Sessions[1].Tabs, catalogue.RemoteCatalogTab{ID: "tab-1"})
89+ if original.Sessions[1].Tabs == nil || len(original.Sessions[1].Tabs) != 0 {
90+ t.Fatalf("Clone() shares tab memory with the original: %+v", original.Sessions[1].Tabs)
91+ }
8592 }
@@ -72,7 +72,7 @@ func TestRemoteDirectorySnapshotCloneIsDefensive(t *testing.T) {
72 func TestRemoteHostSnapshotCloneIsDefensive(t *testing.T) {72 func TestRemoteHostSnapshotCloneIsDefensive(t *testing.T) {
73 original := RemoteHostSnapshot{73 original := RemoteHostSnapshot{
74 Endpoint: "user@arch",74 Endpoint: "user@arch",
75- Sessions: []catalogue.RemoteCatalogSession{{Name: "work"}},75+ Sessions: []catalogue.RemoteCatalogSession{{Name: "work"}, {Name: "empty", Tabs: []catalogue.RemoteCatalogTab{}}},
76 }76 }
77 cloned := original.Clone()77 cloned := original.Clone()
78 cloned.Sessions[0].Name = "mutated"78 cloned.Sessions[0].Name = "mutated"
@@ -82,4 +82,11 @@ func TestRemoteHostSnapshotCloneIsDefensive(t *testing.T) {
82 if (RemoteHostSnapshot{}).Clone().Sessions != nil {82 if (RemoteHostSnapshot{}).Clone().Sessions != nil {
83 t.Fatal("Clone() of a session-less host must stay nil")83 t.Fatal("Clone() of a session-less host must stay nil")
84 }84 }
85+ if cloned.Sessions[1].Tabs == nil {
86+ t.Fatal("Clone() must keep an empty but present tab list")
87+ }
88+ cloned.Sessions[1].Tabs = append(cloned.Sessions[1].Tabs, catalogue.RemoteCatalogTab{ID: "tab-1"})
89+ if original.Sessions[1].Tabs == nil || len(original.Sessions[1].Tabs) != 0 {
90+ t.Fatalf("Clone() shares tab memory with the original: %+v", original.Sessions[1].Tabs)
91+ }
85 }92 }
M internal/usecase/broker/registry.go +184 -42
@@ -7,6 +7,7 @@ import (
77 "fmt"
88 "hash/fnv"
99 "log/slog"
10+ "reflect"
1011 "sort"
1112 "sync"
1213 "sync/atomic"
@@ -44,9 +45,20 @@ type Registry struct {
4445 log *slog.Logger
4546 store *snapshotWriter
4647
48+ // hostStore and authority are the synchronous membership owner. Only this
49+ // registry may mutate the store while it is live; the caller owns Close.
50+ hostStore ports.BrokerHostStore
51+ authority ports.BrokerHosts // guarded by mu; independent of publication revision
52+
4753 current atomic.Pointer[ports.BrokerSnapshot]
4854 mu sync.Mutex
4955 revision ports.BrokerRevision
56+ // revisionExhausted records that this epoch consumed its entire revision
57+ // series. The registry then fails closed: the last valid snapshot stays
58+ // published, nothing is written to the store, and every mutation path
59+ // reports ports.ErrBrokerRevisionExhausted instead of wrapping the series to
60+ // zero or below the newest revision.
61+ revisionExhausted bool
5062 // attempts is the registry-wide monotonic attempt token source. Every
5163 // started observation claims one token, and only the token currently
5264 // recorded as in flight may complete. Tokens are never reused, so a stale
@@ -73,8 +85,16 @@ type Registry struct {
7385 }
7486
7587 // NewRegistry restores a validated durable snapshot under a fresh broker epoch.
76-func NewRegistry(epoch ports.BrokerEpoch, store ports.BrokerSnapshotStore, probe ports.BrokerHostProbe, clock ports.Clock, log *slog.Logger) (*Registry, error) {
77- if epoch == 0 || probe == nil || clock == nil {
88+// The store owns durable membership as well as the advisory snapshot, so
89+// BrokerHostStore is required explicitly: a store that cannot answer
90+// LoadHosts is not a broker host store, and an optional assertion would let a
91+// caller silently run the registry without authoritative membership. A nil
92+// store is refused by the dependency guard, including a typed nil hidden
93+// behind the interface. Until Run returns, callers must not mutate the store
94+// outside Registry.ReplaceHosts. The caller closes the store after the registry
95+// has stopped and flushed its advisory writer.
96+func NewRegistry(epoch ports.BrokerEpoch, store ports.BrokerHostStore, probe ports.BrokerHostProbe, clock ports.Clock, log *slog.Logger) (*Registry, error) {
97+ if epoch == 0 || nilDependency(store) || nilDependency(probe) || nilDependency(clock) {
7898 return nil, errors.New("broker: invalid registry dependencies")
7999 }
80100 if log == nil {
@@ -88,20 +108,68 @@ func NewRegistry(epoch ports.BrokerEpoch, store ports.BrokerSnapshotStore, probe
88108 freshFor: defaultFreshFor, retryBase: defaultRetryBase, retryMax: defaultRetryLimit,
89109 jitter: jittered,
90110 }
91- if store != nil {
92- r.store = newSnapshotWriter(store, log)
93- snapshot, err := store.Load()
94- if err != nil {
95- return nil, err
96- }
97- if err := r.restore(snapshot); err != nil {
98- return nil, err
99- }
111+ r.store = newSnapshotWriter(store, log)
112+ snapshot, err := store.Load()
113+ if err != nil {
114+ return nil, err
115+ }
116+ if err := r.restore(snapshot); err != nil {
117+ return nil, err
100118 }
119+ hosts, err := store.LoadHosts()
120+ if err != nil {
121+ return nil, err
122+ }
123+ if err := r.projectMembership(hosts); err != nil {
124+ return nil, err
125+ }
126+ r.hostStore = store
127+ r.authority = ports.BrokerHosts{Revision: hosts.Revision, Hosts: append([]ports.BrokerHostRecord(nil), hosts.Hosts...)}
101128 r.publishLocked(false)
102129 return r, nil
103130 }
104131
132+// nilDependency reports whether a required dependency is absent, including a
133+// typed nil pointer stored behind a non-nil interface. Comparing the interface
134+// to nil alone would admit a typed nil and defer the failure to the first
135+// method call, which panics; the reflection check keeps the dependency guard
136+// total for every nil shape.
137+func nilDependency(dependency any) bool {
138+ if dependency == nil {
139+ return true
140+ }
141+ value := reflect.ValueOf(dependency)
142+ switch value.Kind() {
143+ case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Ptr, reflect.Slice:
144+ return value.IsNil()
145+ default:
146+ return false
147+ }
148+}
149+
150+// projectMembership intersects durable membership with the observations a
151+// previous process left behind. Durable membership is authoritative:
152+// snapshots cannot resurrect removed hosts or omit never-observed
153+// registrations. A restored observation is reused only when it belongs to the
154+// exact same registration identity; any other registration starts unknown and
155+// carries no inventory.
156+func (r *Registry) projectMembership(hosts ports.BrokerHosts) error {
157+ if err := hosts.Validate(); err != nil {
158+ return err
159+ }
160+ next := make(map[string]ports.RemoteHostSnapshot, len(hosts.Hosts))
161+ for _, record := range hosts.Hosts {
162+ registration := record.Registration
163+ restored, present := r.hosts[registration.Endpoint]
164+ if !present || !restored.Registration.Equal(registration) {
165+ restored = ports.RemoteHostSnapshot{Endpoint: registration.Endpoint, Registration: registration, Availability: domain.RemoteAvailabilityUnknown}
166+ }
167+ next[registration.Endpoint] = restored
168+ }
169+ r.hosts = next
170+ return nil
171+}
172+
105173 // restore adopts a validated durable snapshot. A zero-epoch snapshot is only
106174 // the empty placeholder a store returns before its first write: any record
107175 // without an epoch is rejected instead of silently dropped. A persisted
@@ -151,12 +219,53 @@ func (r *Registry) Subscribe() ports.BrokerSubscription {
151219 return s
152220 }
153221
154-// SetHosts atomically replaces configured membership. An unchanged membership
155-// is a no-op: it neither republishes nor restarts an in-flight observation.
222+// ReplaceHosts durably replaces membership and explicit policy using the loaded
223+// authority revision. Success means membership is committed, not that advisory
224+// observations have flushed. Store errors (including stale authority) are
225+// returned without changing the projection, probe attempts, or local token.
226+// A settled run reports ports.ErrBrokerRegistryClosed and an exhausted revision
227+// series reports ports.ErrBrokerRevisionExhausted; both refuse the mutation
228+// before the store CAS, so a refusal never changes durable authority either.
229+// Conflicts are not retried or refreshed implicitly: reconcile by reopening the
230+// owner. Callers must supply fresh incarnations when removing and re-adding.
231+// Even identical records perform CAS so stale authority cannot report success.
232+// The synchronous write holds mu; Snapshot remains lock-free, but scheduling
233+// waits for membership durability. Observation writes remain asynchronous.
234+func (r *Registry) ReplaceHosts(records []ports.BrokerHostRecord) error {
235+ records = append([]ports.BrokerHostRecord(nil), records...)
236+ if err := ports.ValidateBrokerHostRecords(records); err != nil {
237+ return err
238+ }
239+ next := make(map[string]ports.RemoteHostSnapshot, len(records))
240+ for _, record := range records {
241+ reg := record.Registration
242+ next[reg.Endpoint] = ports.RemoteHostSnapshot{Endpoint: reg.Endpoint, Registration: reg, Availability: domain.RemoteAvailabilityUnknown}
243+ }
244+ r.mu.Lock()
245+ defer r.mu.Unlock()
246+ if r.running.Load() == runStopped {
247+ return fmt.Errorf("broker: membership replacement: %w", ports.ErrBrokerRegistryClosed)
248+ }
249+ if r.revisionExhaustedLocked() {
250+ return fmt.Errorf("broker: membership replacement: %w", ports.ErrBrokerRevisionExhausted)
251+ }
252+ if err := r.hostStore.ReplaceHosts(r.authority.Revision, records); err != nil {
253+ return fmt.Errorf("broker: replace hosts: %w", err)
254+ }
255+ r.authority = ports.BrokerHosts{Revision: r.authority.Revision + 1, Hosts: records}
256+ r.setHostsLocked(next)
257+ return nil
258+}
259+
260+// setHosts is projection-only; it is private so callers cannot bypass durable
261+// membership and policy. Observation tests use it without claiming durability.
262+// An unchanged membership is a no-op: it neither republishes nor restarts an
263+// in-flight observation. An exhausted revision series is refused with
264+// ports.ErrBrokerRevisionExhausted before the projection changes.
156265 // Removed registrations retire bounded tombstones so a stale completion can
157266 // never revive them; replaced or re-added endpoints are fenced by the
158267 // registry attempt token instead.
159-func (r *Registry) SetHosts(registrations []domain.RemoteRegistration) error {
268+func (r *Registry) setHosts(registrations []domain.RemoteRegistration) error {
160269 if len(registrations) > ports.BrokerMaxHosts {
161270 return fmt.Errorf("broker: %d host registrations exceed the limit of %d", len(registrations), ports.BrokerMaxHosts)
162271 }
@@ -172,9 +281,19 @@ func (r *Registry) SetHosts(registrations []domain.RemoteRegistration) error {
172281 }
173282
174283 r.mu.Lock()
284+ defer r.mu.Unlock()
285+ if r.revisionExhaustedLocked() {
286+ return fmt.Errorf("broker: set hosts: %w", ports.ErrBrokerRevisionExhausted)
287+ }
288+ r.setHostsLocked(next)
289+ return nil
290+}
291+
292+// setHostsLocked applies only a projection; the public path commits authority
293+// first. The caller holds mu through both steps, fencing probe completions.
294+func (r *Registry) setHostsLocked(next map[string]ports.RemoteHostSnapshot) {
175295 if registrationsUnchanged(r.hosts, next) {
176- r.mu.Unlock()
177- return nil
296+ return
178297 }
179298 for endpoint, current := range r.hosts {
180299 candidate, present := next[endpoint]
@@ -207,9 +326,7 @@ func (r *Registry) SetHosts(registrations []domain.RemoteRegistration) error {
207326 }
208327 r.hosts = next
209328 r.publishLocked(true)
210- r.mu.Unlock()
211329 r.hint()
212- return nil
213330 }
214331
215332 // cancelInflightLocked cancels the in-flight attempt for one endpoint and
@@ -301,11 +418,9 @@ func (r *Registry) settle() {
301418 if cleared {
302419 r.publishLocked(true)
303420 }
304- r.mu.Unlock()
305- if r.store != nil {
306- r.store.close()
307- }
308421 r.running.Store(runStopped)
422+ r.mu.Unlock()
423+ r.store.close()
309424 }
310425
311426 // probeAttempt is one admitted observation: the registry-wide token that
@@ -392,9 +507,6 @@ func (r *Registry) apply(result probeResult) {
392507
393508 kind, availability, failed := observationOutcome(result)
394509 if !failed {
395- // Reachable: adopt the observed inventory, but registry-owned
396- // accounting (success time, attempt counters, failure episode) stays
397- // local.
398510 observed := result.snapshot.Clone()
399511 observed.Endpoint = result.endpoint
400512 observed.Registration = result.registration
@@ -405,9 +517,17 @@ func (r *Registry) apply(result probeResult) {
405517 observed.ConsecutiveFailures = 0
406518 observed.LastFailure = domain.RemoteFailure{}
407519 observed.FailureEpisode = current.FailureEpisode
408- r.hosts[result.endpoint] = observed
409- r.publishLocked(true)
410- return
520+ // A projection the durable store would reject is an invalid response,
521+ // not an observation: publishing it would leave the registry valid but
522+ // permanently unpersisted. The store applies the same rule.
523+ if err := ports.ValidateDurableHostProjection(observed); err != nil {
524+ r.log.Debug("broker: observation is not persistable", "endpoint", result.endpoint, "err", err)
525+ kind, availability, failed = domain.RemoteFailureInvalidResponse, domain.RemoteAvailabilityInvalidResponse, true
526+ } else {
527+ r.hosts[result.endpoint] = observed
528+ r.publishLocked(true)
529+ return
530+ }
411531 }
412532 // Every non-reachable outcome is a typed failure on the capped retry
413533 // cadence: the last success, last-known inventory, and failure episode
@@ -425,7 +545,10 @@ func (r *Registry) apply(result probeResult) {
425545
426546 // observationOutcome classifies a completed observation. It reports
427547 // failed=false only for a confirmed reachable result; every other outcome
428-// carries the typed failure kind and the availability to publish.
548+// carries the typed failure kind and the availability to publish. The caller
549+// additionally checks the adopted projection against the durable rules
550+// (ports.ValidateDurableHostProjection), which cover the session bound and
551+// catalogue validity for the exact incarnation and success time.
429552 func observationOutcome(result probeResult) (domain.RemoteFailureKind, domain.RemoteAvailability, bool) {
430553 if result.err != nil {
431554 kind := domain.RemoteFailureTransport
@@ -434,13 +557,6 @@ func observationOutcome(result probeResult) (domain.RemoteFailureKind, domain.Re
434557 }
435558 return kind, availabilityFor(kind), true
436559 }
437- // A projection carrying more sessions than the publication bound is a
438- // malformed response: publishing it would produce a snapshot that fails
439- // BrokerSnapshot.Validate. Classify the whole projection as an invalid
440- // response so the last-known inventory stays authoritative.
441- if len(result.snapshot.Sessions) > ports.BrokerMaxSessionsPerHost {
442- return domain.RemoteFailureInvalidResponse, domain.RemoteAvailabilityInvalidResponse, true
443- }
444560 switch result.snapshot.Availability {
445561 case domain.RemoteAvailabilityReachable:
446562 return domain.RemoteFailureNone, domain.RemoteAvailabilityReachable, false
@@ -548,12 +664,11 @@ func registrationsUnchanged(current, next map[string]ports.RemoteHostSnapshot) b
548664 }
549665
550666 // retireLocked records the tombstone for one removed registration at the
551-// revision of the publication that carries it.
667+// revision of the publication that carries it. Mutation paths refuse an
668+// exhausted series before retiring, so the retirement revision can never wrap
669+// to zero or fall below the last published one.
552670 func (r *Registry) retireLocked(registration domain.RemoteRegistration) {
553671 retired := r.revision + 1
554- if retired == 0 {
555- retired = 1
556- }
557672 r.tombstones[registration.Endpoint] = ports.BrokerHostTombstone{
558673 Endpoint: registration.Endpoint,
559674 Registration: registration,
@@ -562,6 +677,21 @@ func (r *Registry) retireLocked(registration domain.RemoteRegistration) {
562677 r.pruneTombstonesLocked()
563678 }
564679
680+// revisionExhaustedLocked reports whether this epoch has no revision left to
681+// publish, latching the exhausted state on first detection. Callers must hold
682+// r.mu.
683+func (r *Registry) revisionExhaustedLocked() bool {
684+ if r.revisionExhausted {
685+ return true
686+ }
687+ if r.revision != ^ports.BrokerRevision(0) {
688+ return false
689+ }
690+ r.revisionExhausted = true
691+ r.log.Error("broker: revision series exhausted; refusing further publication", "epoch", r.epoch)
692+ return true
693+}
694+
565695 // pruneTombstonesLocked keeps only the newest BrokerMaxTombstones retirements
566696 // so an endpoint that churns can never grow the published set without bound.
567697 func (r *Registry) pruneTombstonesLocked() {
@@ -604,6 +734,14 @@ func (r *Registry) tombstonesLocked() []ports.BrokerHostTombstone {
604734 }
605735
606736 func (r *Registry) publishLocked(persist bool) {
737+ // Fail closed on an exhausted series: a wrapped revision would be zero, and
738+ // a wrapped series would publish revisions below the newest one, either of
739+ // which silently freezes durable persistence. The last valid snapshot stays
740+ // published, nothing is stored, and no subscriber is woken for a publication
741+ // that did not happen.
742+ if r.revisionExhaustedLocked() {
743+ return
744+ }
607745 r.revision++
608746 hosts := make([]ports.RemoteHostSnapshot, 0, len(r.hosts))
609747 for _, host := range r.hosts {
@@ -612,7 +750,7 @@ func (r *Registry) publishLocked(persist bool) {
612750 sort.Slice(hosts, func(i, j int) bool { return hosts[i].Endpoint < hosts[j].Endpoint })
613751 snapshot := ports.BrokerSnapshot{Epoch: r.epoch, Revision: r.revision, Hosts: hosts, Removed: r.tombstonesLocked()}
614752 r.current.Store(&snapshot)
615- if persist && r.store != nil {
753+ if persist {
616754 // enqueue only stages an immutable publication: rendering the durable
617755 // copy and every store call happen on the writer goroutine, never
618756 // under the registry lock and never on the probe schedule.
@@ -625,9 +763,13 @@ func (r *Registry) publishLocked(persist bool) {
625763
626764 // durable returns the persistable copy of a publication, rendered by the
627765 // writer goroutine. Checking is transient in-flight state and never reaches
628-// the durable snapshot.
766+// the durable snapshot. Retirement tombstones are process-local fencing state:
767+// they retire registrations only for this process lifetime and a restart never
768+// adopts them, so persisting them would leak stale authority into durable
769+// state. Removed is therefore cleared before persistence.
629770 func durable(snapshot ports.BrokerSnapshot) ports.BrokerSnapshot {
630771 out := snapshot.Clone()
772+ out.Removed = nil
631773 for i := range out.Hosts {
632774 out.Hosts[i].Checking = false
633775 }
@@ -7,6 +7,7 @@ import (
7 "fmt"7 "fmt"
8 "hash/fnv"8 "hash/fnv"
9 "log/slog"9 "log/slog"
10+ "reflect"
10 "sort"11 "sort"
11 "sync"12 "sync"
12 "sync/atomic"13 "sync/atomic"
@@ -44,9 +45,20 @@ type Registry struct {
44 log *slog.Logger45 log *slog.Logger
45 store *snapshotWriter46 store *snapshotWriter
46 47
48+ // hostStore and authority are the synchronous membership owner. Only this
49+ // registry may mutate the store while it is live; the caller owns Close.
50+ hostStore ports.BrokerHostStore
51+ authority ports.BrokerHosts // guarded by mu; independent of publication revision
52+
47 current atomic.Pointer[ports.BrokerSnapshot]53 current atomic.Pointer[ports.BrokerSnapshot]
48 mu sync.Mutex54 mu sync.Mutex
49 revision ports.BrokerRevision55 revision ports.BrokerRevision
56+ // revisionExhausted records that this epoch consumed its entire revision
57+ // series. The registry then fails closed: the last valid snapshot stays
58+ // published, nothing is written to the store, and every mutation path
59+ // reports ports.ErrBrokerRevisionExhausted instead of wrapping the series to
60+ // zero or below the newest revision.
61+ revisionExhausted bool
50 // attempts is the registry-wide monotonic attempt token source. Every62 // attempts is the registry-wide monotonic attempt token source. Every
51 // started observation claims one token, and only the token currently63 // started observation claims one token, and only the token currently
52 // recorded as in flight may complete. Tokens are never reused, so a stale64 // recorded as in flight may complete. Tokens are never reused, so a stale
@@ -73,8 +85,16 @@ type Registry struct {
73 }85 }
74 86
75 // NewRegistry restores a validated durable snapshot under a fresh broker epoch.87 // NewRegistry restores a validated durable snapshot under a fresh broker epoch.
76-func NewRegistry(epoch ports.BrokerEpoch, store ports.BrokerSnapshotStore, probe ports.BrokerHostProbe, clock ports.Clock, log *slog.Logger) (*Registry, error) {88+// The store owns durable membership as well as the advisory snapshot, so
77- if epoch == 0 || probe == nil || clock == nil {89+// BrokerHostStore is required explicitly: a store that cannot answer
90+// LoadHosts is not a broker host store, and an optional assertion would let a
91+// caller silently run the registry without authoritative membership. A nil
92+// store is refused by the dependency guard, including a typed nil hidden
93+// behind the interface. Until Run returns, callers must not mutate the store
94+// outside Registry.ReplaceHosts. The caller closes the store after the registry
95+// has stopped and flushed its advisory writer.
96+func NewRegistry(epoch ports.BrokerEpoch, store ports.BrokerHostStore, probe ports.BrokerHostProbe, clock ports.Clock, log *slog.Logger) (*Registry, error) {
97+ if epoch == 0 || nilDependency(store) || nilDependency(probe) || nilDependency(clock) {
78 return nil, errors.New("broker: invalid registry dependencies")98 return nil, errors.New("broker: invalid registry dependencies")
79 }99 }
80 if log == nil {100 if log == nil {
@@ -88,20 +108,68 @@ func NewRegistry(epoch ports.BrokerEpoch, store ports.BrokerSnapshotStore, probe
88 freshFor: defaultFreshFor, retryBase: defaultRetryBase, retryMax: defaultRetryLimit,108 freshFor: defaultFreshFor, retryBase: defaultRetryBase, retryMax: defaultRetryLimit,
89 jitter: jittered,109 jitter: jittered,
90 }110 }
91- if store != nil {111+ r.store = newSnapshotWriter(store, log)
92- r.store = newSnapshotWriter(store, log)112+ snapshot, err := store.Load()
93- snapshot, err := store.Load()113+ if err != nil {
94- if err != nil {114+ return nil, err
95- return nil, err115+ }
96- }116+ if err := r.restore(snapshot); err != nil {
97- if err := r.restore(snapshot); err != nil {117+ return nil, err
98- return nil, err
99- }
100 }118 }
119+ hosts, err := store.LoadHosts()
120+ if err != nil {
121+ return nil, err
122+ }
123+ if err := r.projectMembership(hosts); err != nil {
124+ return nil, err
125+ }
126+ r.hostStore = store
127+ r.authority = ports.BrokerHosts{Revision: hosts.Revision, Hosts: append([]ports.BrokerHostRecord(nil), hosts.Hosts...)}
101 r.publishLocked(false)128 r.publishLocked(false)
102 return r, nil129 return r, nil
103 }130 }
104 131
132+// nilDependency reports whether a required dependency is absent, including a
133+// typed nil pointer stored behind a non-nil interface. Comparing the interface
134+// to nil alone would admit a typed nil and defer the failure to the first
135+// method call, which panics; the reflection check keeps the dependency guard
136+// total for every nil shape.
137+func nilDependency(dependency any) bool {
138+ if dependency == nil {
139+ return true
140+ }
141+ value := reflect.ValueOf(dependency)
142+ switch value.Kind() {
143+ case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Ptr, reflect.Slice:
144+ return value.IsNil()
145+ default:
146+ return false
147+ }
148+}
149+
150+// projectMembership intersects durable membership with the observations a
151+// previous process left behind. Durable membership is authoritative:
152+// snapshots cannot resurrect removed hosts or omit never-observed
153+// registrations. A restored observation is reused only when it belongs to the
154+// exact same registration identity; any other registration starts unknown and
155+// carries no inventory.
156+func (r *Registry) projectMembership(hosts ports.BrokerHosts) error {
157+ if err := hosts.Validate(); err != nil {
158+ return err
159+ }
160+ next := make(map[string]ports.RemoteHostSnapshot, len(hosts.Hosts))
161+ for _, record := range hosts.Hosts {
162+ registration := record.Registration
163+ restored, present := r.hosts[registration.Endpoint]
164+ if !present || !restored.Registration.Equal(registration) {
165+ restored = ports.RemoteHostSnapshot{Endpoint: registration.Endpoint, Registration: registration, Availability: domain.RemoteAvailabilityUnknown}
166+ }
167+ next[registration.Endpoint] = restored
168+ }
169+ r.hosts = next
170+ return nil
171+}
172+
105 // restore adopts a validated durable snapshot. A zero-epoch snapshot is only173 // restore adopts a validated durable snapshot. A zero-epoch snapshot is only
106 // the empty placeholder a store returns before its first write: any record174 // the empty placeholder a store returns before its first write: any record
107 // without an epoch is rejected instead of silently dropped. A persisted175 // without an epoch is rejected instead of silently dropped. A persisted
@@ -151,12 +219,53 @@ func (r *Registry) Subscribe() ports.BrokerSubscription {
151 return s219 return s
152 }220 }
153 221
154-// SetHosts atomically replaces configured membership. An unchanged membership222+// ReplaceHosts durably replaces membership and explicit policy using the loaded
155-// is a no-op: it neither republishes nor restarts an in-flight observation.223+// authority revision. Success means membership is committed, not that advisory
224+// observations have flushed. Store errors (including stale authority) are
225+// returned without changing the projection, probe attempts, or local token.
226+// A settled run reports ports.ErrBrokerRegistryClosed and an exhausted revision
227+// series reports ports.ErrBrokerRevisionExhausted; both refuse the mutation
228+// before the store CAS, so a refusal never changes durable authority either.
229+// Conflicts are not retried or refreshed implicitly: reconcile by reopening the
230+// owner. Callers must supply fresh incarnations when removing and re-adding.
231+// Even identical records perform CAS so stale authority cannot report success.
232+// The synchronous write holds mu; Snapshot remains lock-free, but scheduling
233+// waits for membership durability. Observation writes remain asynchronous.
234+func (r *Registry) ReplaceHosts(records []ports.BrokerHostRecord) error {
235+ records = append([]ports.BrokerHostRecord(nil), records...)
236+ if err := ports.ValidateBrokerHostRecords(records); err != nil {
237+ return err
238+ }
239+ next := make(map[string]ports.RemoteHostSnapshot, len(records))
240+ for _, record := range records {
241+ reg := record.Registration
242+ next[reg.Endpoint] = ports.RemoteHostSnapshot{Endpoint: reg.Endpoint, Registration: reg, Availability: domain.RemoteAvailabilityUnknown}
243+ }
244+ r.mu.Lock()
245+ defer r.mu.Unlock()
246+ if r.running.Load() == runStopped {
247+ return fmt.Errorf("broker: membership replacement: %w", ports.ErrBrokerRegistryClosed)
248+ }
249+ if r.revisionExhaustedLocked() {
250+ return fmt.Errorf("broker: membership replacement: %w", ports.ErrBrokerRevisionExhausted)
251+ }
252+ if err := r.hostStore.ReplaceHosts(r.authority.Revision, records); err != nil {
253+ return fmt.Errorf("broker: replace hosts: %w", err)
254+ }
255+ r.authority = ports.BrokerHosts{Revision: r.authority.Revision + 1, Hosts: records}
256+ r.setHostsLocked(next)
257+ return nil
258+}
259+
260+// setHosts is projection-only; it is private so callers cannot bypass durable
261+// membership and policy. Observation tests use it without claiming durability.
262+// An unchanged membership is a no-op: it neither republishes nor restarts an
263+// in-flight observation. An exhausted revision series is refused with
264+// ports.ErrBrokerRevisionExhausted before the projection changes.
156 // Removed registrations retire bounded tombstones so a stale completion can265 // Removed registrations retire bounded tombstones so a stale completion can
157 // never revive them; replaced or re-added endpoints are fenced by the266 // never revive them; replaced or re-added endpoints are fenced by the
158 // registry attempt token instead.267 // registry attempt token instead.
159-func (r *Registry) SetHosts(registrations []domain.RemoteRegistration) error {268+func (r *Registry) setHosts(registrations []domain.RemoteRegistration) error {
160 if len(registrations) > ports.BrokerMaxHosts {269 if len(registrations) > ports.BrokerMaxHosts {
161 return fmt.Errorf("broker: %d host registrations exceed the limit of %d", len(registrations), ports.BrokerMaxHosts)270 return fmt.Errorf("broker: %d host registrations exceed the limit of %d", len(registrations), ports.BrokerMaxHosts)
162 }271 }
@@ -172,9 +281,19 @@ func (r *Registry) SetHosts(registrations []domain.RemoteRegistration) error {
172 }281 }
173 282
174 r.mu.Lock()283 r.mu.Lock()
284+ defer r.mu.Unlock()
285+ if r.revisionExhaustedLocked() {
286+ return fmt.Errorf("broker: set hosts: %w", ports.ErrBrokerRevisionExhausted)
287+ }
288+ r.setHostsLocked(next)
289+ return nil
290+}
291+
292+// setHostsLocked applies only a projection; the public path commits authority
293+// first. The caller holds mu through both steps, fencing probe completions.
294+func (r *Registry) setHostsLocked(next map[string]ports.RemoteHostSnapshot) {
175 if registrationsUnchanged(r.hosts, next) {295 if registrationsUnchanged(r.hosts, next) {
176- r.mu.Unlock()296+ return
177- return nil
178 }297 }
179 for endpoint, current := range r.hosts {298 for endpoint, current := range r.hosts {
180 candidate, present := next[endpoint]299 candidate, present := next[endpoint]
@@ -207,9 +326,7 @@ func (r *Registry) SetHosts(registrations []domain.RemoteRegistration) error {
207 }326 }
208 r.hosts = next327 r.hosts = next
209 r.publishLocked(true)328 r.publishLocked(true)
210- r.mu.Unlock()
211 r.hint()329 r.hint()
212- return nil
213 }330 }
214 331
215 // cancelInflightLocked cancels the in-flight attempt for one endpoint and332 // cancelInflightLocked cancels the in-flight attempt for one endpoint and
@@ -301,11 +418,9 @@ func (r *Registry) settle() {
301 if cleared {418 if cleared {
302 r.publishLocked(true)419 r.publishLocked(true)
303 }420 }
304- r.mu.Unlock()
305- if r.store != nil {
306- r.store.close()
307- }
308 r.running.Store(runStopped)421 r.running.Store(runStopped)
422+ r.mu.Unlock()
423+ r.store.close()
309 }424 }
310 425
311 // probeAttempt is one admitted observation: the registry-wide token that426 // probeAttempt is one admitted observation: the registry-wide token that
@@ -392,9 +507,6 @@ func (r *Registry) apply(result probeResult) {
392 507
393 kind, availability, failed := observationOutcome(result)508 kind, availability, failed := observationOutcome(result)
394 if !failed {509 if !failed {
395- // Reachable: adopt the observed inventory, but registry-owned
396- // accounting (success time, attempt counters, failure episode) stays
397- // local.
398 observed := result.snapshot.Clone()510 observed := result.snapshot.Clone()
399 observed.Endpoint = result.endpoint511 observed.Endpoint = result.endpoint
400 observed.Registration = result.registration512 observed.Registration = result.registration
@@ -405,9 +517,17 @@ func (r *Registry) apply(result probeResult) {
405 observed.ConsecutiveFailures = 0517 observed.ConsecutiveFailures = 0
406 observed.LastFailure = domain.RemoteFailure{}518 observed.LastFailure = domain.RemoteFailure{}
407 observed.FailureEpisode = current.FailureEpisode519 observed.FailureEpisode = current.FailureEpisode
408- r.hosts[result.endpoint] = observed520+ // A projection the durable store would reject is an invalid response,
409- r.publishLocked(true)521+ // not an observation: publishing it would leave the registry valid but
410- return522+ // permanently unpersisted. The store applies the same rule.
523+ if err := ports.ValidateDurableHostProjection(observed); err != nil {
524+ r.log.Debug("broker: observation is not persistable", "endpoint", result.endpoint, "err", err)
525+ kind, availability, failed = domain.RemoteFailureInvalidResponse, domain.RemoteAvailabilityInvalidResponse, true
526+ } else {
527+ r.hosts[result.endpoint] = observed
528+ r.publishLocked(true)
529+ return
530+ }
411 }531 }
412 // Every non-reachable outcome is a typed failure on the capped retry532 // Every non-reachable outcome is a typed failure on the capped retry
413 // cadence: the last success, last-known inventory, and failure episode533 // cadence: the last success, last-known inventory, and failure episode
@@ -425,7 +545,10 @@ func (r *Registry) apply(result probeResult) {
425 545
426 // observationOutcome classifies a completed observation. It reports546 // observationOutcome classifies a completed observation. It reports
427 // failed=false only for a confirmed reachable result; every other outcome547 // failed=false only for a confirmed reachable result; every other outcome
428-// carries the typed failure kind and the availability to publish.548+// carries the typed failure kind and the availability to publish. The caller
549+// additionally checks the adopted projection against the durable rules
550+// (ports.ValidateDurableHostProjection), which cover the session bound and
551+// catalogue validity for the exact incarnation and success time.
429 func observationOutcome(result probeResult) (domain.RemoteFailureKind, domain.RemoteAvailability, bool) {552 func observationOutcome(result probeResult) (domain.RemoteFailureKind, domain.RemoteAvailability, bool) {
430 if result.err != nil {553 if result.err != nil {
431 kind := domain.RemoteFailureTransport554 kind := domain.RemoteFailureTransport
@@ -434,13 +557,6 @@ func observationOutcome(result probeResult) (domain.RemoteFailureKind, domain.Re
434 }557 }
435 return kind, availabilityFor(kind), true558 return kind, availabilityFor(kind), true
436 }559 }
437- // A projection carrying more sessions than the publication bound is a
438- // malformed response: publishing it would produce a snapshot that fails
439- // BrokerSnapshot.Validate. Classify the whole projection as an invalid
440- // response so the last-known inventory stays authoritative.
441- if len(result.snapshot.Sessions) > ports.BrokerMaxSessionsPerHost {
442- return domain.RemoteFailureInvalidResponse, domain.RemoteAvailabilityInvalidResponse, true
443- }
444 switch result.snapshot.Availability {560 switch result.snapshot.Availability {
445 case domain.RemoteAvailabilityReachable:561 case domain.RemoteAvailabilityReachable:
446 return domain.RemoteFailureNone, domain.RemoteAvailabilityReachable, false562 return domain.RemoteFailureNone, domain.RemoteAvailabilityReachable, false
@@ -548,12 +664,11 @@ func registrationsUnchanged(current, next map[string]ports.RemoteHostSnapshot) b
548 }664 }
549 665
550 // retireLocked records the tombstone for one removed registration at the666 // retireLocked records the tombstone for one removed registration at the
551-// revision of the publication that carries it.667+// revision of the publication that carries it. Mutation paths refuse an
668+// exhausted series before retiring, so the retirement revision can never wrap
669+// to zero or fall below the last published one.
552 func (r *Registry) retireLocked(registration domain.RemoteRegistration) {670 func (r *Registry) retireLocked(registration domain.RemoteRegistration) {
553 retired := r.revision + 1671 retired := r.revision + 1
554- if retired == 0 {
555- retired = 1
556- }
557 r.tombstones[registration.Endpoint] = ports.BrokerHostTombstone{672 r.tombstones[registration.Endpoint] = ports.BrokerHostTombstone{
558 Endpoint: registration.Endpoint,673 Endpoint: registration.Endpoint,
559 Registration: registration,674 Registration: registration,
@@ -562,6 +677,21 @@ func (r *Registry) retireLocked(registration domain.RemoteRegistration) {
562 r.pruneTombstonesLocked()677 r.pruneTombstonesLocked()
563 }678 }
564 679
680+// revisionExhaustedLocked reports whether this epoch has no revision left to
681+// publish, latching the exhausted state on first detection. Callers must hold
682+// r.mu.
683+func (r *Registry) revisionExhaustedLocked() bool {
684+ if r.revisionExhausted {
685+ return true
686+ }
687+ if r.revision != ^ports.BrokerRevision(0) {
688+ return false
689+ }
690+ r.revisionExhausted = true
691+ r.log.Error("broker: revision series exhausted; refusing further publication", "epoch", r.epoch)
692+ return true
693+}
694+
565 // pruneTombstonesLocked keeps only the newest BrokerMaxTombstones retirements695 // pruneTombstonesLocked keeps only the newest BrokerMaxTombstones retirements
566 // so an endpoint that churns can never grow the published set without bound.696 // so an endpoint that churns can never grow the published set without bound.
567 func (r *Registry) pruneTombstonesLocked() {697 func (r *Registry) pruneTombstonesLocked() {
@@ -604,6 +734,14 @@ func (r *Registry) tombstonesLocked() []ports.BrokerHostTombstone {
604 }734 }
605 735
606 func (r *Registry) publishLocked(persist bool) {736 func (r *Registry) publishLocked(persist bool) {
737+ // Fail closed on an exhausted series: a wrapped revision would be zero, and
738+ // a wrapped series would publish revisions below the newest one, either of
739+ // which silently freezes durable persistence. The last valid snapshot stays
740+ // published, nothing is stored, and no subscriber is woken for a publication
741+ // that did not happen.
742+ if r.revisionExhaustedLocked() {
743+ return
744+ }
607 r.revision++745 r.revision++
608 hosts := make([]ports.RemoteHostSnapshot, 0, len(r.hosts))746 hosts := make([]ports.RemoteHostSnapshot, 0, len(r.hosts))
609 for _, host := range r.hosts {747 for _, host := range r.hosts {
@@ -612,7 +750,7 @@ func (r *Registry) publishLocked(persist bool) {
612 sort.Slice(hosts, func(i, j int) bool { return hosts[i].Endpoint < hosts[j].Endpoint })750 sort.Slice(hosts, func(i, j int) bool { return hosts[i].Endpoint < hosts[j].Endpoint })
613 snapshot := ports.BrokerSnapshot{Epoch: r.epoch, Revision: r.revision, Hosts: hosts, Removed: r.tombstonesLocked()}751 snapshot := ports.BrokerSnapshot{Epoch: r.epoch, Revision: r.revision, Hosts: hosts, Removed: r.tombstonesLocked()}
614 r.current.Store(&snapshot)752 r.current.Store(&snapshot)
615- if persist && r.store != nil {753+ if persist {
616 // enqueue only stages an immutable publication: rendering the durable754 // enqueue only stages an immutable publication: rendering the durable
617 // copy and every store call happen on the writer goroutine, never755 // copy and every store call happen on the writer goroutine, never
618 // under the registry lock and never on the probe schedule.756 // under the registry lock and never on the probe schedule.
@@ -625,9 +763,13 @@ func (r *Registry) publishLocked(persist bool) {
625 763
626 // durable returns the persistable copy of a publication, rendered by the764 // durable returns the persistable copy of a publication, rendered by the
627 // writer goroutine. Checking is transient in-flight state and never reaches765 // writer goroutine. Checking is transient in-flight state and never reaches
628-// the durable snapshot.766+// the durable snapshot. Retirement tombstones are process-local fencing state:
767+// they retire registrations only for this process lifetime and a restart never
768+// adopts them, so persisting them would leak stale authority into durable
769+// state. Removed is therefore cleared before persistence.
629 func durable(snapshot ports.BrokerSnapshot) ports.BrokerSnapshot {770 func durable(snapshot ports.BrokerSnapshot) ports.BrokerSnapshot {
630 out := snapshot.Clone()771 out := snapshot.Clone()
772+ out.Removed = nil
631 for i := range out.Hosts {773 for i := range out.Hosts {
632 out.Hosts[i].Checking = false774 out.Hosts[i].Checking = false
633 }775 }
A internal/usecase/broker/registry_durable_test.go +165 -0
@@ -0,0 +1,165 @@
1+package broker
2+
3+import (
4+ "context"
5+ "path/filepath"
6+ "testing"
7+ "time"
8+
9+ "github.com/bnema/vev/internal/adapters/brokerstore"
10+ "github.com/bnema/vev/internal/domain"
11+ "github.com/bnema/vev/internal/ports"
12+ portsmocks "github.com/bnema/vev/internal/ports/mocks"
13+ "github.com/stretchr/testify/require"
14+)
15+
16+// Exercise the real CAS and snapshot validator, not the observation-only fake.
17+// Each stage closes and reopens both owners, so membership and observations
18+// must come from disk, including the empty set and a fresh re-added identity.
19+func TestRegistryDurableReplaceHostsRestart(t *testing.T) {
20+ dir := filepath.Join(t.TempDir(), "broker")
21+ first := ports.BrokerHostRecord{Registration: registration(t, "host", 1), Pinned: true, Policy: poolPolicy()}
22+ readded := first
23+ readded.Registration = registration(t, "host", 2)
24+ readded.Pinned, readded.Learned = false, true
25+ readded.Policy.Trust = "new-trust"
26+ var previous []ports.BrokerHostRecord
27+ for i, stage := range []struct {
28+ name string
29+ records []ports.BrokerHostRecord
30+ }{
31+ {"add", []ports.BrokerHostRecord{first}}, {"remove", nil}, {"re-add", []ports.BrokerHostRecord{readded}}, {"restart", []ports.BrokerHostRecord{readded}},
32+ } {
33+ t.Run(stage.name, func(t *testing.T) {
34+ store, err := brokerstore.OpenOffline(brokerstore.Options{Dir: dir})
35+ require.NoError(t, err)
36+ defer store.Close()
37+ r, err := NewRegistry(ports.BrokerEpoch(i+1), store, newTestProbe(1), newManualClock(time.Unix(100, 0)), nil)
38+ require.NoError(t, err)
39+ defer r.settle()
40+ require.Equal(t, previous, r.authority.Hosts)
41+ require.Len(t, r.Snapshot().Hosts, len(previous))
42+ if len(previous) > 0 {
43+ require.True(t, r.Snapshot().Hosts[0].InventoryKnown, "advisory observation must also survive restart")
44+ }
45+ require.NoError(t, r.ReplaceHosts(stage.records))
46+ authority, err := store.LoadHosts()
47+ require.NoError(t, err)
48+ require.Equal(t, r.authority, authority)
49+ require.Equal(t, stage.records, authority.Hosts)
50+ require.Len(t, r.Snapshot().Hosts, len(stage.records))
51+ if len(stage.records) > 0 {
52+ reg := stage.records[0].Registration
53+ ctx, cancel := context.WithCancel(context.Background())
54+ defer cancel()
55+ r.inflight[reg.Endpoint] = &probeAttempt{token: 1, cancel: cancel}
56+ r.apply(probeResult{endpoint: reg.Endpoint, registration: reg, attempt: 1, at: r.clock.Now(), snapshot: ports.RemoteHostSnapshot{Availability: domain.RemoteAvailabilityReachable, InventoryKnown: true}})
57+ require.ErrorIs(t, ctx.Err(), context.Canceled)
58+ }
59+ r.settle() // flush before inspecting persisted projection
60+ snapshot, err := store.Load()
61+ require.NoError(t, err)
62+ require.Equal(t, durable(r.Snapshot()), snapshot)
63+ require.ErrorIs(t, r.ReplaceHosts(stage.records), ports.ErrBrokerRegistryClosed, "shutdown must not acknowledge mutations")
64+ previous = stage.records
65+ })
66+ }
67+}
68+
69+func TestRegistryDurableConflictDoesNotMutateProjection(t *testing.T) {
70+ for _, mode := range []string{"stale revision", "policy without new generation"} {
71+ t.Run(mode, func(t *testing.T) {
72+ store, err := brokerstore.OpenOffline(brokerstore.Options{Dir: filepath.Join(t.TempDir(), "broker")})
73+ require.NoError(t, err)
74+ defer store.Close()
75+ record := ports.BrokerHostRecord{Registration: registration(t, "host", 1), Pinned: true, Policy: poolPolicy()}
76+ require.NoError(t, store.ReplaceHosts(1, []ports.BrokerHostRecord{record}))
77+ r, err := NewRegistry(1, store, newTestProbe(1), newManualClock(time.Unix(100, 0)), nil)
78+ require.NoError(t, err)
79+ defer r.settle()
80+ before := r.Snapshot()
81+ authority := r.authority
82+ ctx, cancel := context.WithCancel(context.Background())
83+ defer cancel()
84+ r.inflight[record.Registration.Endpoint] = &probeAttempt{token: 1, cancel: cancel}
85+ if mode == "stale revision" {
86+ // Simulate an illicit second mutator. Even identical local records must
87+ // CAS rather than returning success from a projection-only no-op.
88+ require.NoError(t, store.ReplaceHosts(authority.Revision, nil))
89+ } else {
90+ record.Policy.Trust = "changed-trust"
91+ }
92+ require.ErrorIs(t, r.ReplaceHosts([]ports.BrokerHostRecord{record}), ports.ErrBrokerHostConflict)
93+ require.Equal(t, before, r.Snapshot())
94+ require.Equal(t, authority, r.authority)
95+ require.NoError(t, ctx.Err(), "failed CAS must not cancel the current observation")
96+ })
97+ }
98+}
99+
100+func TestRegistryReplaceHostsStoreFailureIsSynchronous(t *testing.T) {
101+ store := portsmocks.NewMockBrokerHostStore(t)
102+ store.EXPECT().Load().Return(ports.BrokerSnapshot{}, nil).Once()
103+ store.EXPECT().LoadHosts().Return(ports.BrokerHosts{Revision: 7}, nil).Once()
104+ record := ports.BrokerHostRecord{Registration: registration(t, "host", 1), Pinned: true, Policy: poolPolicy()}
105+ records := []ports.BrokerHostRecord{record}
106+ store.EXPECT().ReplaceHosts(uint64(7), records).Return(context.DeadlineExceeded).Once()
107+ r, err := NewRegistry(1, store, newTestProbe(1), newManualClock(time.Unix(100, 0)), nil)
108+ require.NoError(t, err)
109+ before := r.Snapshot()
110+ require.ErrorIs(t, r.ReplaceHosts(records), context.DeadlineExceeded)
111+ require.Equal(t, before, r.Snapshot())
112+ require.Equal(t, uint64(7), r.authority.Revision)
113+ require.Error(t, r.ReplaceHosts([]ports.BrokerHostRecord{{Registration: record.Registration}}))
114+ require.Equal(t, before, r.Snapshot())
115+}
116+
117+// TestRegistryReplaceHostsAfterShutdownIsTyped pins the closed sentinel: once
118+// the run has settled, a membership mutation is refused with a typed error, so
119+// a caller can tell shutdown apart from a store failure without matching text.
120+func TestRegistryReplaceHostsAfterShutdownIsTyped(t *testing.T) {
121+ store := newTestStore()
122+ r := newTestRegistry(t, 1, store, newTestProbe(1), newManualClock(time.Unix(100, 0)))
123+ record := ports.BrokerHostRecord{Registration: registration(t, "host", 1), Pinned: true, Policy: poolPolicy()}
124+ require.NoError(t, r.setHosts([]domain.RemoteRegistration{record.Registration}))
125+ r.settle()
126+
127+ before := r.Snapshot()
128+ authority := r.authority
129+ err := r.ReplaceHosts([]ports.BrokerHostRecord{record})
130+ require.ErrorIs(t, err, ports.ErrBrokerRegistryClosed)
131+ require.NotErrorIs(t, err, ports.ErrBrokerRevisionExhausted)
132+ require.Equal(t, before, r.Snapshot())
133+ require.Equal(t, authority, r.authority)
134+}
135+
136+// Queued pre-CAS observations may be rejected as stale by the real store, but
137+// the newest post-CAS publication must always flush; churn cannot leave the
138+// registry permanently publishing membership that the store will reject.
139+func TestRegistryDurableChurnFlushesLatestMembership(t *testing.T) {
140+ store, err := brokerstore.OpenOffline(brokerstore.Options{Dir: filepath.Join(t.TempDir(), "broker")})
141+ require.NoError(t, err)
142+ defer store.Close()
143+ r, err := NewRegistry(1, store, newTestProbe(1), newManualClock(time.Unix(100, 0)), nil)
144+ require.NoError(t, err)
145+ defer r.settle()
146+ for i := byte(1); i <= 20; i++ {
147+ record := ports.BrokerHostRecord{Registration: registration(t, "host", i), Pinned: true, Policy: poolPolicy()}
148+ require.NoError(t, r.ReplaceHosts([]ports.BrokerHostRecord{record}))
149+ // A policy change is admitted only with a new registration generation.
150+ record.Registration.Generation++
151+ record.Policy.Trust = "replacement-trust"
152+ require.NoError(t, r.ReplaceHosts([]ports.BrokerHostRecord{record}))
153+ require.Equal(t, record.Registration, r.Snapshot().Hosts[0].Registration)
154+ if i < 20 {
155+ require.NoError(t, r.ReplaceHosts(nil))
156+ }
157+ }
158+ r.settle()
159+ snapshot, err := store.Load()
160+ require.NoError(t, err)
161+ require.Equal(t, durable(r.Snapshot()), snapshot)
162+ authority, err := store.LoadHosts()
163+ require.NoError(t, err)
164+ require.Equal(t, r.authority, authority)
165+}
@@ -0,0 +1,165 @@
1+package broker
2+
3+import (
4+ "context"
5+ "path/filepath"
6+ "testing"
7+ "time"
8+
9+ "github.com/bnema/vev/internal/adapters/brokerstore"
10+ "github.com/bnema/vev/internal/domain"
11+ "github.com/bnema/vev/internal/ports"
12+ portsmocks "github.com/bnema/vev/internal/ports/mocks"
13+ "github.com/stretchr/testify/require"
14+)
15+
16+// Exercise the real CAS and snapshot validator, not the observation-only fake.
17+// Each stage closes and reopens both owners, so membership and observations
18+// must come from disk, including the empty set and a fresh re-added identity.
19+func TestRegistryDurableReplaceHostsRestart(t *testing.T) {
20+ dir := filepath.Join(t.TempDir(), "broker")
21+ first := ports.BrokerHostRecord{Registration: registration(t, "host", 1), Pinned: true, Policy: poolPolicy()}
22+ readded := first
23+ readded.Registration = registration(t, "host", 2)
24+ readded.Pinned, readded.Learned = false, true
25+ readded.Policy.Trust = "new-trust"
26+ var previous []ports.BrokerHostRecord
27+ for i, stage := range []struct {
28+ name string
29+ records []ports.BrokerHostRecord
30+ }{
31+ {"add", []ports.BrokerHostRecord{first}}, {"remove", nil}, {"re-add", []ports.BrokerHostRecord{readded}}, {"restart", []ports.BrokerHostRecord{readded}},
32+ } {
33+ t.Run(stage.name, func(t *testing.T) {
34+ store, err := brokerstore.OpenOffline(brokerstore.Options{Dir: dir})
35+ require.NoError(t, err)
36+ defer store.Close()
37+ r, err := NewRegistry(ports.BrokerEpoch(i+1), store, newTestProbe(1), newManualClock(time.Unix(100, 0)), nil)
38+ require.NoError(t, err)
39+ defer r.settle()
40+ require.Equal(t, previous, r.authority.Hosts)
41+ require.Len(t, r.Snapshot().Hosts, len(previous))
42+ if len(previous) > 0 {
43+ require.True(t, r.Snapshot().Hosts[0].InventoryKnown, "advisory observation must also survive restart")
44+ }
45+ require.NoError(t, r.ReplaceHosts(stage.records))
46+ authority, err := store.LoadHosts()
47+ require.NoError(t, err)
48+ require.Equal(t, r.authority, authority)
49+ require.Equal(t, stage.records, authority.Hosts)
50+ require.Len(t, r.Snapshot().Hosts, len(stage.records))
51+ if len(stage.records) > 0 {
52+ reg := stage.records[0].Registration
53+ ctx, cancel := context.WithCancel(context.Background())
54+ defer cancel()
55+ r.inflight[reg.Endpoint] = &probeAttempt{token: 1, cancel: cancel}
56+ r.apply(probeResult{endpoint: reg.Endpoint, registration: reg, attempt: 1, at: r.clock.Now(), snapshot: ports.RemoteHostSnapshot{Availability: domain.RemoteAvailabilityReachable, InventoryKnown: true}})
57+ require.ErrorIs(t, ctx.Err(), context.Canceled)
58+ }
59+ r.settle() // flush before inspecting persisted projection
60+ snapshot, err := store.Load()
61+ require.NoError(t, err)
62+ require.Equal(t, durable(r.Snapshot()), snapshot)
63+ require.ErrorIs(t, r.ReplaceHosts(stage.records), ports.ErrBrokerRegistryClosed, "shutdown must not acknowledge mutations")
64+ previous = stage.records
65+ })
66+ }
67+}
68+
69+func TestRegistryDurableConflictDoesNotMutateProjection(t *testing.T) {
70+ for _, mode := range []string{"stale revision", "policy without new generation"} {
71+ t.Run(mode, func(t *testing.T) {
72+ store, err := brokerstore.OpenOffline(brokerstore.Options{Dir: filepath.Join(t.TempDir(), "broker")})
73+ require.NoError(t, err)
74+ defer store.Close()
75+ record := ports.BrokerHostRecord{Registration: registration(t, "host", 1), Pinned: true, Policy: poolPolicy()}
76+ require.NoError(t, store.ReplaceHosts(1, []ports.BrokerHostRecord{record}))
77+ r, err := NewRegistry(1, store, newTestProbe(1), newManualClock(time.Unix(100, 0)), nil)
78+ require.NoError(t, err)
79+ defer r.settle()
80+ before := r.Snapshot()
81+ authority := r.authority
82+ ctx, cancel := context.WithCancel(context.Background())
83+ defer cancel()
84+ r.inflight[record.Registration.Endpoint] = &probeAttempt{token: 1, cancel: cancel}
85+ if mode == "stale revision" {
86+ // Simulate an illicit second mutator. Even identical local records must
87+ // CAS rather than returning success from a projection-only no-op.
88+ require.NoError(t, store.ReplaceHosts(authority.Revision, nil))
89+ } else {
90+ record.Policy.Trust = "changed-trust"
91+ }
92+ require.ErrorIs(t, r.ReplaceHosts([]ports.BrokerHostRecord{record}), ports.ErrBrokerHostConflict)
93+ require.Equal(t, before, r.Snapshot())
94+ require.Equal(t, authority, r.authority)
95+ require.NoError(t, ctx.Err(), "failed CAS must not cancel the current observation")
96+ })
97+ }
98+}
99+
100+func TestRegistryReplaceHostsStoreFailureIsSynchronous(t *testing.T) {
101+ store := portsmocks.NewMockBrokerHostStore(t)
102+ store.EXPECT().Load().Return(ports.BrokerSnapshot{}, nil).Once()
103+ store.EXPECT().LoadHosts().Return(ports.BrokerHosts{Revision: 7}, nil).Once()
104+ record := ports.BrokerHostRecord{Registration: registration(t, "host", 1), Pinned: true, Policy: poolPolicy()}
105+ records := []ports.BrokerHostRecord{record}
106+ store.EXPECT().ReplaceHosts(uint64(7), records).Return(context.DeadlineExceeded).Once()
107+ r, err := NewRegistry(1, store, newTestProbe(1), newManualClock(time.Unix(100, 0)), nil)
108+ require.NoError(t, err)
109+ before := r.Snapshot()
110+ require.ErrorIs(t, r.ReplaceHosts(records), context.DeadlineExceeded)
111+ require.Equal(t, before, r.Snapshot())
112+ require.Equal(t, uint64(7), r.authority.Revision)
113+ require.Error(t, r.ReplaceHosts([]ports.BrokerHostRecord{{Registration: record.Registration}}))
114+ require.Equal(t, before, r.Snapshot())
115+}
116+
117+// TestRegistryReplaceHostsAfterShutdownIsTyped pins the closed sentinel: once
118+// the run has settled, a membership mutation is refused with a typed error, so
119+// a caller can tell shutdown apart from a store failure without matching text.
120+func TestRegistryReplaceHostsAfterShutdownIsTyped(t *testing.T) {
121+ store := newTestStore()
122+ r := newTestRegistry(t, 1, store, newTestProbe(1), newManualClock(time.Unix(100, 0)))
123+ record := ports.BrokerHostRecord{Registration: registration(t, "host", 1), Pinned: true, Policy: poolPolicy()}
124+ require.NoError(t, r.setHosts([]domain.RemoteRegistration{record.Registration}))
125+ r.settle()
126+
127+ before := r.Snapshot()
128+ authority := r.authority
129+ err := r.ReplaceHosts([]ports.BrokerHostRecord{record})
130+ require.ErrorIs(t, err, ports.ErrBrokerRegistryClosed)
131+ require.NotErrorIs(t, err, ports.ErrBrokerRevisionExhausted)
132+ require.Equal(t, before, r.Snapshot())
133+ require.Equal(t, authority, r.authority)
134+}
135+
136+// Queued pre-CAS observations may be rejected as stale by the real store, but
137+// the newest post-CAS publication must always flush; churn cannot leave the
138+// registry permanently publishing membership that the store will reject.
139+func TestRegistryDurableChurnFlushesLatestMembership(t *testing.T) {
140+ store, err := brokerstore.OpenOffline(brokerstore.Options{Dir: filepath.Join(t.TempDir(), "broker")})
141+ require.NoError(t, err)
142+ defer store.Close()
143+ r, err := NewRegistry(1, store, newTestProbe(1), newManualClock(time.Unix(100, 0)), nil)
144+ require.NoError(t, err)
145+ defer r.settle()
146+ for i := byte(1); i <= 20; i++ {
147+ record := ports.BrokerHostRecord{Registration: registration(t, "host", i), Pinned: true, Policy: poolPolicy()}
148+ require.NoError(t, r.ReplaceHosts([]ports.BrokerHostRecord{record}))
149+ // A policy change is admitted only with a new registration generation.
150+ record.Registration.Generation++
151+ record.Policy.Trust = "replacement-trust"
152+ require.NoError(t, r.ReplaceHosts([]ports.BrokerHostRecord{record}))
153+ require.Equal(t, record.Registration, r.Snapshot().Hosts[0].Registration)
154+ if i < 20 {
155+ require.NoError(t, r.ReplaceHosts(nil))
156+ }
157+ }
158+ r.settle()
159+ snapshot, err := store.Load()
160+ require.NoError(t, err)
161+ require.Equal(t, durable(r.Snapshot()), snapshot)
162+ authority, err := store.LoadHosts()
163+ require.NoError(t, err)
164+ require.Equal(t, r.authority, authority)
165+}
A internal/usecase/broker/registry_store_test.go +116 -0
@@ -0,0 +1,116 @@
1+package broker
2+
3+import (
4+ "errors"
5+ "testing"
6+ "time"
7+
8+ "github.com/bnema/vev/internal/domain"
9+ "github.com/bnema/vev/internal/ports"
10+ portsmocks "github.com/bnema/vev/internal/ports/mocks"
11+ "github.com/stretchr/testify/require"
12+)
13+
14+func TestRegistryRestoresAuthoritativeMembership(t *testing.T) {
15+ for _, mode := range []string{"removed", "replaced", "never observed", "matching"} {
16+ t.Run(mode, func(t *testing.T) {
17+ old := registration(t, "host", 1)
18+ current := old
19+ store := portsmocks.NewMockBrokerHostStore(t)
20+ snapshot := ports.BrokerSnapshot{Epoch: 1, Revision: 9, Hosts: []ports.RemoteHostSnapshot{{Endpoint: "host", Registration: old, Availability: domain.RemoteAvailabilityReachable, InventoryKnown: true}}}
21+ hosts := ports.BrokerHosts{Revision: 4}
22+ if mode == "replaced" {
23+ current.Generation++
24+ }
25+ if mode != "removed" {
26+ hosts.Hosts = []ports.BrokerHostRecord{{Registration: current, Pinned: true, Policy: poolPolicy()}}
27+ }
28+ if mode == "never observed" {
29+ snapshot = ports.BrokerSnapshot{}
30+ }
31+ store.EXPECT().Load().Return(snapshot, nil).Once()
32+ store.EXPECT().LoadHosts().Return(hosts, nil).Once()
33+ r, err := NewRegistry(2, store, newTestProbe(1), newManualClock(time.Unix(100, 0)), nil)
34+ require.NoError(t, err)
35+ got := r.Snapshot()
36+ require.Equal(t, ports.BrokerEpoch(2), got.Epoch)
37+ require.Equal(t, ports.BrokerRevision(1), got.Revision)
38+ if mode == "removed" {
39+ require.Empty(t, got.Hosts)
40+ return
41+ }
42+ require.Len(t, got.Hosts, 1)
43+ require.Equal(t, current, got.Hosts[0].Registration)
44+ require.Equal(t, mode == "matching", got.Hosts[0].InventoryKnown)
45+ })
46+ }
47+}
48+
49+// TestRegistryRejectsAuthorityAndLoadFailures covers the negative store seam:
50+// NewRegistry requires BrokerHostStore explicitly, so a load error, an
51+// authority error, or authority that fails validation is fatal instead of
52+// being skipped as an optional capability.
53+func TestRegistryRejectsAuthorityAndLoadFailures(t *testing.T) {
54+ clock := newManualClock(time.Unix(100, 0))
55+ reg := registration(t, "host", 1)
56+ loadErr := errors.New("snapshot load failed")
57+ authorityErr := errors.New("authority load failed")
58+ invalidAuthority := map[string]ports.BrokerHosts{
59+ "missing revision": {Hosts: []ports.BrokerHostRecord{{Registration: reg, Pinned: true, Policy: poolPolicy()}}},
60+ "duplicate endpoint": {Revision: 1, Hosts: []ports.BrokerHostRecord{
61+ {Registration: reg, Pinned: true, Policy: poolPolicy()},
62+ {Registration: reg, Learned: true, Policy: poolPolicy()},
63+ }},
64+ "unanchored record": {Revision: 1, Hosts: []ports.BrokerHostRecord{{Registration: reg, Policy: poolPolicy()}}},
65+ "zero policy": {Revision: 1, Hosts: []ports.BrokerHostRecord{{Registration: reg, Pinned: true}}},
66+ }
67+ for name, hosts := range invalidAuthority {
68+ t.Run(name, func(t *testing.T) {
69+ store := portsmocks.NewMockBrokerHostStore(t)
70+ store.EXPECT().Load().Return(ports.BrokerSnapshot{}, nil).Once()
71+ store.EXPECT().LoadHosts().Return(hosts, nil).Once()
72+ registry, err := NewRegistry(1, store, newTestProbe(1), clock, nil)
73+ require.Error(t, err)
74+ require.Nil(t, registry)
75+ })
76+ }
77+ t.Run("snapshot load error", func(t *testing.T) {
78+ store := portsmocks.NewMockBrokerHostStore(t)
79+ store.EXPECT().Load().Return(ports.BrokerSnapshot{}, loadErr).Once()
80+ registry, err := NewRegistry(1, store, newTestProbe(1), clock, nil)
81+ require.ErrorIs(t, err, loadErr)
82+ require.Nil(t, registry)
83+ })
84+ t.Run("authority load error", func(t *testing.T) {
85+ store := portsmocks.NewMockBrokerHostStore(t)
86+ store.EXPECT().Load().Return(ports.BrokerSnapshot{}, nil).Once()
87+ store.EXPECT().LoadHosts().Return(ports.BrokerHosts{}, authorityErr).Once()
88+ registry, err := NewRegistry(1, store, newTestProbe(1), clock, nil)
89+ require.ErrorIs(t, err, authorityErr)
90+ require.Nil(t, registry)
91+ })
92+}
93+
94+// TestRegistryRestoreIsReadOnlyAndPublicationLocal pins the revision series: the
95+// durable authority revision is store-local and never seeds the publication
96+// series, and construction writes nothing back.
97+func TestRegistryRestoreIsReadOnlyAndPublicationLocal(t *testing.T) {
98+ clock := newManualClock(time.Unix(100, 0))
99+ reg := registration(t, "host", 1)
100+ store := portsmocks.NewMockBrokerHostStore(t)
101+ store.EXPECT().Load().Return(ports.BrokerSnapshot{Epoch: 5, Revision: 4096, Hosts: []ports.RemoteHostSnapshot{{
102+ Endpoint: reg.Endpoint, Registration: reg, Availability: domain.RemoteAvailabilityReachable, InventoryKnown: true,
103+ }}}, nil).Once()
104+ store.EXPECT().LoadHosts().Return(ports.BrokerHosts{Revision: 99, Hosts: []ports.BrokerHostRecord{{Registration: reg, Pinned: true, Policy: poolPolicy()}}}, nil).Once()
105+
106+ r, err := NewRegistry(6, store, newTestProbe(1), clock, nil)
107+ require.NoError(t, err)
108+ snapshot := r.Snapshot()
109+ require.Equal(t, ports.BrokerRevision(1), snapshot.Revision)
110+ require.Equal(t, ports.BrokerEpoch(6), snapshot.Epoch)
111+ // Restoring reuses the matching observation but never republishes it: a
112+ // fresh process adopts authority and rewrites only when it observes.
113+ require.Len(t, snapshot.Hosts, 1)
114+ require.True(t, snapshot.Hosts[0].InventoryKnown)
115+ require.Equal(t, domain.RemoteAvailabilityReachable, snapshot.Hosts[0].Availability)
116+}
@@ -0,0 +1,116 @@
1+package broker
2+
3+import (
4+ "errors"
5+ "testing"
6+ "time"
7+
8+ "github.com/bnema/vev/internal/domain"
9+ "github.com/bnema/vev/internal/ports"
10+ portsmocks "github.com/bnema/vev/internal/ports/mocks"
11+ "github.com/stretchr/testify/require"
12+)
13+
14+func TestRegistryRestoresAuthoritativeMembership(t *testing.T) {
15+ for _, mode := range []string{"removed", "replaced", "never observed", "matching"} {
16+ t.Run(mode, func(t *testing.T) {
17+ old := registration(t, "host", 1)
18+ current := old
19+ store := portsmocks.NewMockBrokerHostStore(t)
20+ snapshot := ports.BrokerSnapshot{Epoch: 1, Revision: 9, Hosts: []ports.RemoteHostSnapshot{{Endpoint: "host", Registration: old, Availability: domain.RemoteAvailabilityReachable, InventoryKnown: true}}}
21+ hosts := ports.BrokerHosts{Revision: 4}
22+ if mode == "replaced" {
23+ current.Generation++
24+ }
25+ if mode != "removed" {
26+ hosts.Hosts = []ports.BrokerHostRecord{{Registration: current, Pinned: true, Policy: poolPolicy()}}
27+ }
28+ if mode == "never observed" {
29+ snapshot = ports.BrokerSnapshot{}
30+ }
31+ store.EXPECT().Load().Return(snapshot, nil).Once()
32+ store.EXPECT().LoadHosts().Return(hosts, nil).Once()
33+ r, err := NewRegistry(2, store, newTestProbe(1), newManualClock(time.Unix(100, 0)), nil)
34+ require.NoError(t, err)
35+ got := r.Snapshot()
36+ require.Equal(t, ports.BrokerEpoch(2), got.Epoch)
37+ require.Equal(t, ports.BrokerRevision(1), got.Revision)
38+ if mode == "removed" {
39+ require.Empty(t, got.Hosts)
40+ return
41+ }
42+ require.Len(t, got.Hosts, 1)
43+ require.Equal(t, current, got.Hosts[0].Registration)
44+ require.Equal(t, mode == "matching", got.Hosts[0].InventoryKnown)
45+ })
46+ }
47+}
48+
49+// TestRegistryRejectsAuthorityAndLoadFailures covers the negative store seam:
50+// NewRegistry requires BrokerHostStore explicitly, so a load error, an
51+// authority error, or authority that fails validation is fatal instead of
52+// being skipped as an optional capability.
53+func TestRegistryRejectsAuthorityAndLoadFailures(t *testing.T) {
54+ clock := newManualClock(time.Unix(100, 0))
55+ reg := registration(t, "host", 1)
56+ loadErr := errors.New("snapshot load failed")
57+ authorityErr := errors.New("authority load failed")
58+ invalidAuthority := map[string]ports.BrokerHosts{
59+ "missing revision": {Hosts: []ports.BrokerHostRecord{{Registration: reg, Pinned: true, Policy: poolPolicy()}}},
60+ "duplicate endpoint": {Revision: 1, Hosts: []ports.BrokerHostRecord{
61+ {Registration: reg, Pinned: true, Policy: poolPolicy()},
62+ {Registration: reg, Learned: true, Policy: poolPolicy()},
63+ }},
64+ "unanchored record": {Revision: 1, Hosts: []ports.BrokerHostRecord{{Registration: reg, Policy: poolPolicy()}}},
65+ "zero policy": {Revision: 1, Hosts: []ports.BrokerHostRecord{{Registration: reg, Pinned: true}}},
66+ }
67+ for name, hosts := range invalidAuthority {
68+ t.Run(name, func(t *testing.T) {
69+ store := portsmocks.NewMockBrokerHostStore(t)
70+ store.EXPECT().Load().Return(ports.BrokerSnapshot{}, nil).Once()
71+ store.EXPECT().LoadHosts().Return(hosts, nil).Once()
72+ registry, err := NewRegistry(1, store, newTestProbe(1), clock, nil)
73+ require.Error(t, err)
74+ require.Nil(t, registry)
75+ })
76+ }
77+ t.Run("snapshot load error", func(t *testing.T) {
78+ store := portsmocks.NewMockBrokerHostStore(t)
79+ store.EXPECT().Load().Return(ports.BrokerSnapshot{}, loadErr).Once()
80+ registry, err := NewRegistry(1, store, newTestProbe(1), clock, nil)
81+ require.ErrorIs(t, err, loadErr)
82+ require.Nil(t, registry)
83+ })
84+ t.Run("authority load error", func(t *testing.T) {
85+ store := portsmocks.NewMockBrokerHostStore(t)
86+ store.EXPECT().Load().Return(ports.BrokerSnapshot{}, nil).Once()
87+ store.EXPECT().LoadHosts().Return(ports.BrokerHosts{}, authorityErr).Once()
88+ registry, err := NewRegistry(1, store, newTestProbe(1), clock, nil)
89+ require.ErrorIs(t, err, authorityErr)
90+ require.Nil(t, registry)
91+ })
92+}
93+
94+// TestRegistryRestoreIsReadOnlyAndPublicationLocal pins the revision series: the
95+// durable authority revision is store-local and never seeds the publication
96+// series, and construction writes nothing back.
97+func TestRegistryRestoreIsReadOnlyAndPublicationLocal(t *testing.T) {
98+ clock := newManualClock(time.Unix(100, 0))
99+ reg := registration(t, "host", 1)
100+ store := portsmocks.NewMockBrokerHostStore(t)
101+ store.EXPECT().Load().Return(ports.BrokerSnapshot{Epoch: 5, Revision: 4096, Hosts: []ports.RemoteHostSnapshot{{
102+ Endpoint: reg.Endpoint, Registration: reg, Availability: domain.RemoteAvailabilityReachable, InventoryKnown: true,
103+ }}}, nil).Once()
104+ store.EXPECT().LoadHosts().Return(ports.BrokerHosts{Revision: 99, Hosts: []ports.BrokerHostRecord{{Registration: reg, Pinned: true, Policy: poolPolicy()}}}, nil).Once()
105+
106+ r, err := NewRegistry(6, store, newTestProbe(1), clock, nil)
107+ require.NoError(t, err)
108+ snapshot := r.Snapshot()
109+ require.Equal(t, ports.BrokerRevision(1), snapshot.Revision)
110+ require.Equal(t, ports.BrokerEpoch(6), snapshot.Epoch)
111+ // Restoring reuses the matching observation but never republishes it: a
112+ // fresh process adopts authority and rewrites only when it observes.
113+ require.Len(t, snapshot.Hosts, 1)
114+ require.True(t, snapshot.Hosts[0].InventoryKnown)
115+ require.Equal(t, domain.RemoteAvailabilityReachable, snapshot.Hosts[0].Availability)
116+}
M internal/usecase/broker/registry_test.go +388 -87
@@ -3,6 +3,7 @@ package broker
33 import (
44 "bytes"
55 "context"
6+ "crypto/sha256"
67 "errors"
78 "fmt"
89 "log/slog"
@@ -165,11 +166,17 @@ func testLogger() (*slog.Logger, *syncBuffer) {
165166 }
166167
167168 // testStore is a race-safe in-memory durable store whose writes signal a
168-// capacity-one channel so tests wait on a write instead of on wall time.
169+// capacity-one channel so tests wait on a write instead of on wall time. It
170+// implements the whole BrokerHostStore seam: authority defaults to the hosts of
171+// the restored snapshot, so a fixture that only names a durable snapshot still
172+// keeps exercising the rule that observations intersect authoritative
173+// membership.
169174 type testStore struct {
170175 mu sync.Mutex
171176 loaded ports.BrokerSnapshot
172177 loadErr error
178+ hosts ports.BrokerHosts
179+ hostsErr error
173180 stored ports.BrokerSnapshot
174181 storeErr error
175182 writes int
@@ -185,6 +192,41 @@ func (s *testStore) Load() (ports.BrokerSnapshot, error) {
185192 return s.loaded.Clone(), nil
186193 }
187194
195+func (s *testStore) LoadHosts() (ports.BrokerHosts, error) {
196+ s.mu.Lock()
197+ defer s.mu.Unlock()
198+ if s.hostsErr != nil {
199+ return ports.BrokerHosts{}, s.hostsErr
200+ }
201+ if s.hosts.Revision != 0 || len(s.hosts.Hosts) > 0 {
202+ return s.hosts, nil
203+ }
204+ authority := ports.BrokerHosts{Revision: 1}
205+ for _, host := range s.loaded.Hosts {
206+ authority.Hosts = append(authority.Hosts, ports.BrokerHostRecord{Registration: host.Registration, Pinned: true, Policy: poolPolicy()})
207+ }
208+ return authority, nil
209+}
210+
211+func (s *testStore) ReplaceHosts(expected uint64, hosts []ports.BrokerHostRecord) error {
212+ s.mu.Lock()
213+ defer s.mu.Unlock()
214+ if s.hostsErr != nil {
215+ return s.hostsErr
216+ }
217+ if s.hosts.Revision == 0 {
218+ s.hosts.Revision = 1
219+ }
220+ if expected != s.hosts.Revision {
221+ return ports.ErrBrokerHostConflict
222+ }
223+ s.hosts.Hosts = append([]ports.BrokerHostRecord(nil), hosts...)
224+ s.hosts.Revision++
225+ return nil
226+}
227+
228+func (s *testStore) Close() error { return nil }
229+
188230 func (s *testStore) Store(snapshot ports.BrokerSnapshot) error {
189231 s.mu.Lock()
190232 s.stored = snapshot.Clone()
@@ -251,6 +293,14 @@ func newBlockingStore() *blockingStore {
251293
252294 func (s *blockingStore) Load() (ports.BrokerSnapshot, error) { return ports.BrokerSnapshot{}, nil }
253295
296+func (s *blockingStore) LoadHosts() (ports.BrokerHosts, error) {
297+ return ports.BrokerHosts{Revision: 1}, nil
298+}
299+
300+func (s *blockingStore) ReplaceHosts(uint64, []ports.BrokerHostRecord) error { return nil }
301+
302+func (s *blockingStore) Close() error { return nil }
303+
254304 func (s *blockingStore) Store(snapshot ports.BrokerSnapshot) error {
255305 s.mu.Lock()
256306 s.revisions = append(s.revisions, snapshot.Revision)
@@ -273,6 +323,34 @@ func (s *blockingStore) writtenRevisions() []ports.BrokerRevision {
273323 return append([]ports.BrokerRevision(nil), s.revisions...)
274324 }
275325
326+// blockingCASStore parks the first durable membership CAS until it is released,
327+// so a test can prove publication and Snapshot never wait behind membership
328+// durability. Load, LoadHosts, and Store keep the race-safe testStore behavior,
329+// and every later CAS is admitted without blocking.
330+type blockingCASStore struct {
331+ testStore
332+ entered chan struct{}
333+ released chan struct{}
334+ block sync.Once
335+ open sync.Once
336+}
337+
338+func newBlockingCASStore() *blockingCASStore {
339+ return &blockingCASStore{entered: make(chan struct{}, 1), released: make(chan struct{})}
340+}
341+
342+func (s *blockingCASStore) ReplaceHosts(expected uint64, hosts []ports.BrokerHostRecord) error {
343+ s.block.Do(func() {
344+ s.entered <- struct{}{}
345+ <-s.released
346+ })
347+ return s.testStore.ReplaceHosts(expected, hosts)
348+}
349+
350+// releaseCAS lets the parked CAS commit. It is idempotent so a failing test can
351+// unblock the parked mutator from cleanup.
352+func (s *blockingCASStore) releaseCAS() { s.open.Do(func() { close(s.released) }) }
353+
276354 // testProbe scripts one observation per received call: the test owns the
277355 // completion, so a probe never depends on wall time.
278356 type testProbe struct {
@@ -421,22 +499,41 @@ func registration(t *testing.T, endpoint string, marker byte) domain.RemoteRegis
421499 return result
422500 }
423501
424-// hostSessions builds count minimal catalogue sessions for the projection
502+// hostSession builds one catalogue-valid session. The registry applies the
503+// same durable projection rules as the store before adopting an observation, so
504+// a fixture session must carry a non-zero lifecycle identity and a non-nil tab
505+// list to count as durable.
506+func hostSession(name string) catalogue.RemoteCatalogSession {
507+ sum := sha256.Sum256([]byte("lifecycle:" + name))
508+ lifecycle := domain.SessionLifecycleID{}
509+ copy(lifecycle[:], sum[:])
510+ return catalogue.RemoteCatalogSession{
511+ LifecycleID: lifecycle,
512+ Name: name,
513+ State: catalogue.RemoteCatalogSessionUp,
514+ Tabs: []catalogue.RemoteCatalogTab{},
515+ }
516+}
517+
518+// hostSessions builds count minimal catalogue-valid sessions for the projection
425519 // session bound tests.
426520 func hostSessions(count int) []catalogue.RemoteCatalogSession {
427521 sessions := make([]catalogue.RemoteCatalogSession, 0, count)
428522 for index := 0; index < count; index++ {
429- sessions = append(sessions, catalogue.RemoteCatalogSession{
430- Name: fmt.Sprintf("session-%d", index),
431- State: catalogue.RemoteCatalogSessionUp,
432- })
523+ sessions = append(sessions, hostSession(fmt.Sprintf("session-%d", index)))
433524 }
434525 return sessions
435526 }
436527
528+// newTestStore returns the minimal authoritative durable store: a valid
529+// revision series over empty membership. NewRegistry requires a
530+// BrokerHostStore, so every registry fixture names this store explicitly
531+// instead of relying on an optional seam.
532+func newTestStore() *testStore { return &testStore{} }
533+
437534 // newTestRegistry builds a registry with exact scheduling and no logger so a
438535 // failure never depends on jitter.
439-func newTestRegistry(t *testing.T, epoch ports.BrokerEpoch, store ports.BrokerSnapshotStore, probe ports.BrokerHostProbe, clock *manualClock) *Registry {
536+func newTestRegistry(t *testing.T, epoch ports.BrokerEpoch, store ports.BrokerHostStore, probe ports.BrokerHostProbe, clock *manualClock) *Registry {
440537 t.Helper()
441538 registry, err := NewRegistry(epoch, store, probe, clock, nil)
442539 require.NoError(t, err)
@@ -633,11 +730,19 @@ func TestRegistryRestoreRejectsInvalidInputs(t *testing.T) {
633730 clock := newManualClock(time.Unix(100, 0))
634731 reg := registration(t, "example.test", 1)
635732
636- _, err := NewRegistry(0, nil, newTestProbe(1), clock, nil)
733+ _, err := NewRegistry(0, newTestStore(), newTestProbe(1), clock, nil)
734+ require.Error(t, err)
735+ _, err = NewRegistry(1, newTestStore(), nil, clock, nil)
736+ require.Error(t, err)
737+ _, err = NewRegistry(1, newTestStore(), newTestProbe(1), nil, nil)
637738 require.Error(t, err)
638- _, err = NewRegistry(1, nil, nil, clock, nil)
739+
740+ // The store is a required dependency: a literal nil store is rejected, and
741+ // a typed nil behind the interface is rejected without panicking.
742+ _, err = NewRegistry(1, nil, newTestProbe(1), clock, nil)
639743 require.Error(t, err)
640- _, err = NewRegistry(1, nil, newTestProbe(1), nil, nil)
744+ var typedNilStore *testStore
745+ _, err = NewRegistry(1, typedNilStore, newTestProbe(1), clock, nil)
641746 require.Error(t, err)
642747
643748 loadErr := errors.New("load failed")
@@ -698,40 +803,40 @@ func TestRegistryRestoreRejectsPersistedSnapshotFromItsOwnEpoch(t *testing.T) {
698803 }
699804
700805 func TestRegistrySetHostsRejectsInvalidRegistrations(t *testing.T) {
701- registry := newTestRegistry(t, 1, nil, newTestProbe(1), newManualClock(time.Unix(100, 0)))
806+ registry := newTestRegistry(t, 1, newTestStore(), newTestProbe(1), newManualClock(time.Unix(100, 0)))
702807
703808 reg := registration(t, "example.test", 1)
704- require.Error(t, registry.SetHosts([]domain.RemoteRegistration{reg, reg}))
705- require.Error(t, registry.SetHosts([]domain.RemoteRegistration{{Endpoint: reg.Endpoint}}))
809+ require.Error(t, registry.setHosts([]domain.RemoteRegistration{reg, reg}))
810+ require.Error(t, registry.setHosts([]domain.RemoteRegistration{{Endpoint: reg.Endpoint}}))
706811 require.Empty(t, registry.Snapshot().Hosts)
707812 }
708813
709814 func TestRegistrySetHostsRejectsHostLimitWithoutMutation(t *testing.T) {
710- registry := newTestRegistry(t, 1, nil, newTestProbe(1), newManualClock(time.Unix(100, 0)))
815+ registry := newTestRegistry(t, 1, newTestStore(), newTestProbe(1), newManualClock(time.Unix(100, 0)))
711816 reg := registration(t, "example.test", 1)
712- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
817+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
713818
714819 oversized := make([]domain.RemoteRegistration, 0, ports.BrokerMaxHosts+1)
715820 for index := 0; index <= ports.BrokerMaxHosts; index++ {
716821 oversized = append(oversized, registration(t, fmt.Sprintf("host-%d.test", index), byte(index+1)))
717822 }
718823 before := registry.Snapshot()
719- require.Error(t, registry.SetHosts(oversized))
824+ require.Error(t, registry.setHosts(oversized))
720825 // The rejected call mutates nothing: same membership, same revision.
721826 require.Equal(t, before, registry.Snapshot())
722827
723828 // The boundary itself is accepted.
724- require.NoError(t, registry.SetHosts(oversized[:ports.BrokerMaxHosts]))
829+ require.NoError(t, registry.setHosts(oversized[:ports.BrokerMaxHosts]))
725830 require.Len(t, registry.Snapshot().Hosts, ports.BrokerMaxHosts)
726831 }
727832
728833 func TestRegistryUnchangedSetHostsIsNoOp(t *testing.T) {
729834 clock := newManualClock(time.Unix(100, 0))
730835 probe := newTestProbe(2)
731- registry := newTestRegistry(t, 1, nil, probe, clock)
836+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
732837 first := registration(t, "example.test", 1)
733838 second := registration(t, "other.test", 2)
734- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{first, second}))
839+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{first, second}))
735840 startRegistry(t, registry)
736841
737842 // Every configured host is observed exactly once.
@@ -748,8 +853,8 @@ func TestRegistryUnchangedSetHostsIsNoOp(t *testing.T) {
748853
749854 // Identical membership, including in a different order, never republishes
750855 // and never restarts an in-flight observation.
751- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{second, first}))
752- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{first, second}))
856+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{second, first}))
857+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{first, second}))
753858 require.Equal(t, before, registry.Snapshot())
754859 requireNoProbeWithin(t, probe, probeGrace)
755860
@@ -765,18 +870,19 @@ func TestRegistryUnchangedSetHostsIsNoOp(t *testing.T) {
765870 func TestRegistrySnapshotIsImmutable(t *testing.T) {
766871 clock := newManualClock(time.Unix(100, 0))
767872 probe := newTestProbe(1)
768- registry := newTestRegistry(t, 1, nil, probe, clock)
873+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
769874 reg := registration(t, "example.test", 1)
770- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
875+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
771876 startRegistry(t, registry)
772877
773878 call := receiveCall(t, probe)
774879 call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{
775880 Availability: domain.RemoteAvailabilityReachable,
776881 Sessions: []catalogue.RemoteCatalogSession{{
777- Name: "work",
778- State: catalogue.RemoteCatalogSessionUp,
779- Tabs: []catalogue.RemoteCatalogTab{{ID: "tab-1", Name: "shell"}},
882+ LifecycleID: domain.SessionLifecycleID{1},
883+ Name: "work",
884+ State: catalogue.RemoteCatalogSessionUp,
885+ Tabs: []catalogue.RemoteCatalogTab{{ID: "tab-1", Name: "shell"}},
780886 }},
781887 }}
782888 observed := waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {
@@ -799,7 +905,7 @@ func TestRegistrySnapshotIsImmutable(t *testing.T) {
799905 next := receiveCall(t, probe)
800906 next.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{
801907 Availability: domain.RemoteAvailabilityReachable,
802- Sessions: []catalogue.RemoteCatalogSession{{Name: "other", State: catalogue.RemoteCatalogSessionUp}},
908+ Sessions: []catalogue.RemoteCatalogSession{hostSession("other")},
803909 }}
804910 waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {
805911 return !host.Checking && len(host.Sessions) == 1 && host.Sessions[0].Name == "other"
@@ -811,9 +917,9 @@ func TestRegistrySchedulesFreshObservationAfterSuccess(t *testing.T) {
811917 start := time.Unix(100, 0)
812918 clock := newManualClock(start)
813919 probe := newTestProbe(1)
814- registry := newTestRegistry(t, 1, nil, probe, clock)
920+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
815921 reg := registration(t, "example.test", 1)
816- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
922+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
817923 startRegistry(t, registry)
818924
819925 call := receiveCall(t, probe)
@@ -840,9 +946,9 @@ func TestRegistryDemandOverridesFreshSchedule(t *testing.T) {
840946 start := time.Unix(100, 0)
841947 clock := newManualClock(start)
842948 probe := newTestProbe(1)
843- registry := newTestRegistry(t, 1, nil, probe, clock)
949+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
844950 reg := registration(t, "example.test", 1)
845- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
951+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
846952 startRegistry(t, registry)
847953
848954 call := receiveCall(t, probe)
@@ -862,9 +968,9 @@ func TestRegistryCoalescesConcurrentDemand(t *testing.T) {
862968 start := time.Unix(100, 0)
863969 clock := newManualClock(start)
864970 probe := newTestProbe(2)
865- registry := newTestRegistry(t, 1, nil, probe, clock)
971+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
866972 reg := registration(t, "example.test", 1)
867- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
973+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
868974 startRegistry(t, registry)
869975
870976 call := receiveCall(t, probe)
@@ -894,9 +1000,9 @@ func TestRegistryCoalescesConcurrentDemand(t *testing.T) {
8941000 func TestRegistryIgnoresDemandForUnknownEndpoint(t *testing.T) {
8951001 clock := newManualClock(time.Unix(100, 0))
8961002 probe := newTestProbe(1)
897- registry := newTestRegistry(t, 1, nil, probe, clock)
1003+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
8981004 reg := registration(t, "example.test", 1)
899- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
1005+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
9001006 registry.RequestProbe("unknown.test")
9011007 startRegistry(t, registry)
9021008
@@ -913,14 +1019,14 @@ func TestRegistryPublishesReachabilityAndPersists(t *testing.T) {
9131019 store := &testStore{}
9141020 registry := newTestRegistry(t, 3, store, probe, clock)
9151021 reg := registration(t, "example.test", 1)
916- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
1022+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
9171023 startRegistry(t, registry)
9181024
9191025 call := receiveCall(t, probe)
9201026 call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{
9211027 Availability: domain.RemoteAvailabilityReachable,
9221028 InventoryKnown: true,
923- Sessions: []catalogue.RemoteCatalogSession{{Name: "work", State: catalogue.RemoteCatalogSessionUp}},
1029+ Sessions: []catalogue.RemoteCatalogSession{hostSession("work")},
9241030 }}
9251031 snapshot := waitSnapshot(t, registry, func(snapshot ports.BrokerSnapshot) bool {
9261032 host, ok := snapshot.Find(reg.Endpoint)
@@ -996,9 +1102,9 @@ func TestRegistryNonReachableOutcomesUseTypedFailureBackoff(t *testing.T) {
9961102 start := time.Unix(100, 0)
9971103 clock := newManualClock(start)
9981104 probe := newTestProbe(1)
999- registry := newTestRegistry(t, 1, nil, probe, clock)
1105+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
10001106 reg := registration(t, "example.test", 1)
1001- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
1107+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
10021108 startRegistry(t, registry)
10031109
10041110 call := receiveCall(t, probe)
@@ -1027,9 +1133,9 @@ func TestRegistryClassifiesOversizedProbeProjectionAsInvalidResponse(t *testing.
10271133 start := time.Unix(100, 0)
10281134 clock := newManualClock(start)
10291135 probe := newTestProbe(1)
1030- registry := newTestRegistry(t, 1, nil, probe, clock)
1136+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
10311137 reg := registration(t, "example.test", 1)
1032- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
1138+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
10331139 startRegistry(t, registry)
10341140 sub := registry.Subscribe()
10351141 defer sub.Close()
@@ -1038,17 +1144,17 @@ func TestRegistryClassifiesOversizedProbeProjectionAsInvalidResponse(t *testing.
10381144 call := receiveCall(t, probe)
10391145 call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{
10401146 Availability: domain.RemoteAvailabilityReachable,
1041- Sessions: []catalogue.RemoteCatalogSession{{Name: "work", State: catalogue.RemoteCatalogSessionUp}},
1147+ Sessions: []catalogue.RemoteCatalogSession{hostSession("work")},
10421148 }}
10431149 waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {
10441150 return !host.Checking && host.Availability == domain.RemoteAvailabilityReachable
10451151 })
10461152
1047- // A projection carrying one session more than the publication bound is a
1048- // malformed response: it is classified as an invalid response and never
1049- // reaches publication, because such a snapshot fails Validate. The loop
1050- // checks every observed publication, so an invalid snapshot can never pass
1051- // unnoticed.
1153+ // A projection carrying one session more than the durable session bound is
1154+ // a malformed response: it is classified as an invalid response and never
1155+ // reaches publication, because the store enforces the same catalogue rule.
1156+ // The loop checks every observed publication, so an invalid snapshot can
1157+ // never pass unnoticed.
10521158 clock.Advance(defaultFreshFor)
10531159 call = receiveCall(t, probe)
10541160 call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{
@@ -1084,9 +1190,9 @@ func TestRegistryAcceptsProbeProjectionAtSessionBound(t *testing.T) {
10841190 start := time.Unix(100, 0)
10851191 clock := newManualClock(start)
10861192 probe := newTestProbe(1)
1087- registry := newTestRegistry(t, 1, nil, probe, clock)
1193+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
10881194 reg := registration(t, "example.test", 1)
1089- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
1195+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
10901196 startRegistry(t, registry)
10911197
10921198 // The bound itself is valid: exactly BrokerMaxSessionsPerHost sessions are
@@ -1106,20 +1212,106 @@ func TestRegistryAcceptsProbeProjectionAtSessionBound(t *testing.T) {
11061212 require.Len(t, snapshot.Hosts[0].Sessions, ports.BrokerMaxSessionsPerHost)
11071213 }
11081214
1215+// TestRegistryRejectsNonDurableObservation proves the registry and the durable
1216+// store agree on what may be persisted: a reachable projection the store would
1217+// reject is classified as an invalid response, so the last-known inventory
1218+// stays authoritative instead of a snapshot that can never be written.
1219+func TestRegistryRejectsNonDurableObservation(t *testing.T) {
1220+ start := time.Unix(100, 0)
1221+ clock := newManualClock(start)
1222+ probe := newTestProbe(1)
1223+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
1224+ reg := registration(t, "example.test", 1)
1225+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1226+ startRegistry(t, registry)
1227+
1228+ call := receiveCall(t, probe)
1229+ call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{
1230+ Availability: domain.RemoteAvailabilityReachable,
1231+ Sessions: []catalogue.RemoteCatalogSession{hostSession("work")},
1232+ }}
1233+ waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {
1234+ return !host.Checking && host.Availability == domain.RemoteAvailabilityReachable
1235+ })
1236+
1237+ // A probe answer that violates the catalogue rules the store enforces (here
1238+ // a zero lifecycle identity and an absent tab list) is a malformed response.
1239+ clock.Advance(defaultFreshFor)
1240+ call = receiveCall(t, probe)
1241+ call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{
1242+ Availability: domain.RemoteAvailabilityReachable,
1243+ Sessions: []catalogue.RemoteCatalogSession{{Name: "work", State: catalogue.RemoteCatalogSessionUp}},
1244+ }}
1245+ host := waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {
1246+ return !host.Checking && host.ConsecutiveFailures == 1
1247+ })
1248+ require.Equal(t, domain.RemoteAvailabilityInvalidResponse, host.Availability)
1249+ require.Equal(t, domain.RemoteFailureInvalidResponse, host.LastFailure.Kind)
1250+ require.Len(t, host.Sessions, 1)
1251+ require.Equal(t, "work", host.Sessions[0].Name)
1252+ require.Equal(t, start, host.LastSuccess)
1253+ snapshot := registry.Snapshot()
1254+ require.NoError(t, snapshot.Validate())
1255+ require.NoError(t, ports.ValidateDurableHostProjection(snapshot.Hosts[0]))
1256+}
1257+
1258+// TestRegistryRevisionOverflowFailsClosed pins the revision series: zero is not
1259+// a valid revision and revisions never regress, so an exhausted counter fails
1260+// closed instead of wrapping. The last valid snapshot stays published, the
1261+// registry is poisoned, and every mutation path reports a typed error rather
1262+// than emitting revision zero or a revision below the newest one, either of
1263+// which would silently freeze durable persistence.
1264+func TestRegistryRevisionOverflowFailsClosed(t *testing.T) {
1265+ logger, logs := testLogger()
1266+ registry, err := NewRegistry(1, newTestStore(), newTestProbe(1), newManualClock(time.Unix(100, 0)), logger)
1267+ require.NoError(t, err)
1268+ first := registration(t, "example.test", 1)
1269+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{first}))
1270+ before := registry.Snapshot()
1271+ require.NoError(t, before.Validate())
1272+ require.Equal(t, ports.BrokerRevision(2), before.Revision)
1273+ authority := registry.authority
1274+
1275+ // The next publication would overflow the series to zero.
1276+ registry.revision = ^ports.BrokerRevision(0)
1277+
1278+ // Both mutation paths refuse the change before touching the store or the
1279+ // projection, so neither durable authority nor the publication series moves.
1280+ require.ErrorIs(t, registry.setHosts([]domain.RemoteRegistration{registration(t, "other.test", 2)}), ports.ErrBrokerRevisionExhausted)
1281+ require.ErrorIs(t, registry.ReplaceHosts([]ports.BrokerHostRecord{{Registration: first, Pinned: true, Policy: poolPolicy()}}), ports.ErrBrokerRevisionExhausted)
1282+ require.Equal(t, authority, registry.authority)
1283+
1284+ // The publication path itself fails closed too: a direct attempt neither
1285+ // wraps the counter nor replaces the last valid snapshot.
1286+ registry.mu.Lock()
1287+ registry.publishLocked(false)
1288+ registry.mu.Unlock()
1289+
1290+ after := registry.Snapshot()
1291+ require.Equal(t, before, after)
1292+ require.NoError(t, after.Validate())
1293+ require.Greater(t, after.Revision, ports.BrokerRevision(0))
1294+ require.Equal(t, ^ports.BrokerRevision(0), registry.revision)
1295+ require.Equal(t, first, after.Hosts[0].Registration)
1296+ // The refusal is observable: an exhausted series logs instead of failing
1297+ // silently behind a frozen snapshot.
1298+ require.Contains(t, logs.String(), "revision series exhausted")
1299+}
1300+
11091301 func TestRegistryPreservesLastSuccessAndOwnsFailureEpisode(t *testing.T) {
11101302 start := time.Unix(100, 0)
11111303 clock := newManualClock(start)
11121304 probe := newTestProbe(1)
1113- registry := newTestRegistry(t, 1, nil, probe, clock)
1305+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
11141306 reg := registration(t, "example.test", 1)
1115- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
1307+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
11161308 startRegistry(t, registry)
11171309
11181310 // A successful observation owns the success time and opens no episode.
11191311 call := receiveCall(t, probe)
11201312 call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{
11211313 Availability: domain.RemoteAvailabilityReachable,
1122- Sessions: []catalogue.RemoteCatalogSession{{Name: "work", State: catalogue.RemoteCatalogSessionUp}},
1314+ Sessions: []catalogue.RemoteCatalogSession{hostSession("work")},
11231315 }}
11241316 host := waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {
11251317 return !host.Checking && host.Availability == domain.RemoteAvailabilityReachable
@@ -1172,9 +1364,9 @@ func TestRegistryBackoffIsExponentialAndCapped(t *testing.T) {
11721364 start := time.Unix(100, 0)
11731365 clock := newManualClock(start)
11741366 probe := newTestProbe(1)
1175- registry := newTestRegistry(t, 1, nil, probe, clock)
1367+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
11761368 reg := registration(t, "example.test", 1)
1177- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
1369+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
11781370 startRegistry(t, registry)
11791371
11801372 probeErr := errors.New("offline")
@@ -1215,10 +1407,10 @@ func TestRegistryJitterAppliesToRetryAndHealthyRefresh(t *testing.T) {
12151407 start := time.Unix(100, 0)
12161408 clock := newManualClock(start)
12171409 probe := newTestProbe(1)
1218- registry := newTestRegistry(t, 1, nil, probe, clock)
1410+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
12191411 registry.jitter = func(base time.Duration, _ string, _ uint64) time.Duration { return base / 2 }
12201412 reg := registration(t, "example.test", 1)
1221- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
1413+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
12221414 startRegistry(t, registry)
12231415
12241416 call := receiveCall(t, probe)
@@ -1267,20 +1459,20 @@ func TestRegistryFencesStaleCompletionAcrossRegistrationChanges(t *testing.T) {
12671459 replacementAnswer := func(name string) probeAnswer {
12681460 return probeAnswer{snapshot: ports.RemoteHostSnapshot{
12691461 Availability: domain.RemoteAvailabilityReachable,
1270- Sessions: []catalogue.RemoteCatalogSession{{Name: name, State: catalogue.RemoteCatalogSessionUp}},
1462+ Sessions: []catalogue.RemoteCatalogSession{hostSession(name)},
12711463 }}
12721464 }
12731465 assertFenced := func(t *testing.T, current, replacement domain.RemoteRegistration) {
12741466 t.Helper()
12751467 clock := newManualClock(time.Unix(100, 0))
12761468 probe := newTestProbe(2)
1277- registry := newTestRegistry(t, 1, nil, probe, clock)
1278- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{current}))
1469+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
1470+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{current}))
12791471 startRegistry(t, registry)
12801472 currentCall := receiveCall(t, probe)
12811473 require.True(t, currentCall.registration.Equal(current))
12821474
1283- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{replacement}))
1475+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{replacement}))
12841476 replacementCall := receiveCall(t, probe)
12851477 require.True(t, replacementCall.registration.Equal(replacement))
12861478
@@ -1322,9 +1514,9 @@ func TestRegistryFencesStaleCompletionAcrossRegistrationChanges(t *testing.T) {
13221514 func TestRegistryFencesRetiredAttemptsAcrossRemovalAndIdenticalReAdd(t *testing.T) {
13231515 clock := newManualClock(time.Unix(100, 0))
13241516 probe := newTestProbe(2)
1325- registry := newTestRegistry(t, 1, nil, probe, clock)
1517+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
13261518 reg := registration(t, "example.test", 1)
1327- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
1519+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
13281520
13291521 ctx, cancel := context.WithCancel(context.Background())
13301522 defer cancel()
@@ -1338,7 +1530,7 @@ func TestRegistryFencesRetiredAttemptsAcrossRemovalAndIdenticalReAdd(t *testing.
13381530 require.NotZero(t, retired)
13391531
13401532 // Removal retires a bounded tombstone and drops the in-flight token.
1341- require.NoError(t, registry.SetHosts(nil))
1533+ require.NoError(t, registry.setHosts(nil))
13421534 removed := registry.Snapshot()
13431535 require.Empty(t, removed.Hosts)
13441536 require.Len(t, removed.Removed, 1)
@@ -1353,7 +1545,7 @@ func TestRegistryFencesRetiredAttemptsAcrossRemovalAndIdenticalReAdd(t *testing.
13531545
13541546 // Re-adding the identical registration supersedes the tombstone, so only
13551547 // the attempt token can fence the retired completion.
1356- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
1548+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
13571549 require.Empty(t, registry.Snapshot().Removed)
13581550 dispatch()
13591551 freshCall := receiveCall(t, probe)
@@ -1388,8 +1580,8 @@ func TestRegistryRetiresInflightAttemptsOnReplacementAndRemoval(t *testing.T) {
13881580 t.Run(tt.name, func(t *testing.T) {
13891581 clock := newManualClock(time.Unix(100, 0))
13901582 probe := newTestProbe(2)
1391- registry := newTestRegistry(t, 1, nil, probe, clock)
1392- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{current}))
1583+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
1584+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{current}))
13931585 startRegistry(t, registry)
13941586
13951587 call := receiveCall(t, probe)
@@ -1399,7 +1591,7 @@ func TestRegistryRetiresInflightAttemptsOnReplacementAndRemoval(t *testing.T) {
13991591 // attempt: the probe's derived context is cancelled before the
14001592 // replacement can start, so a ctx-honoring probe stops instead of
14011593 // overlapping its successor.
1402- require.NoError(t, registry.SetHosts(tt.next))
1594+ require.NoError(t, registry.setHosts(tt.next))
14031595 select {
14041596 case <-probe.canceled:
14051597 case <-time.After(time.Second):
@@ -1413,11 +1605,11 @@ func TestRegistryChurnKeepsOneActiveProbePerEndpoint(t *testing.T) {
14131605 clock := newManualClock(time.Unix(100, 0))
14141606 probe := newChurnProbe()
14151607 defer probe.openRelease()
1416- registry := newTestRegistry(t, 1, nil, probe, clock)
1608+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
14171609
14181610 first := registration(t, "example.test", 1)
14191611 second := registration(t, "example.test", 2)
1420- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{first}))
1612+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{first}))
14211613
14221614 ctx, cancel := context.WithCancel(context.Background())
14231615 defer cancel()
@@ -1443,7 +1635,7 @@ func TestRegistryChurnKeepsOneActiveProbePerEndpoint(t *testing.T) {
14431635 if round%2 == 0 {
14441636 next = second
14451637 }
1446- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{next}))
1638+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{next}))
14471639 select {
14481640 case <-probe.startedSig:
14491641 case <-time.After(time.Second):
@@ -1465,7 +1657,7 @@ func TestRegistryChurnKeepsOneActiveProbePerEndpoint(t *testing.T) {
14651657 if (worker+round)%2 == 0 {
14661658 next = second
14671659 }
1468- if err := registry.SetHosts([]domain.RemoteRegistration{next}); err != nil {
1660+ if err := registry.setHosts([]domain.RemoteRegistration{next}); err != nil {
14691661 failures[worker] = err
14701662 return
14711663 }
@@ -1503,7 +1695,7 @@ func TestRegistryChurnKeepsOneActiveProbePerEndpoint(t *testing.T) {
15031695 }
15041696
15051697 func TestRegistryPublishesBoundedTombstones(t *testing.T) {
1506- registry := newTestRegistry(t, 1, nil, newTestProbe(1), newManualClock(time.Unix(100, 0)))
1698+ registry := newTestRegistry(t, 1, newTestStore(), newTestProbe(1), newManualClock(time.Unix(100, 0)))
15071699
15081700 // Two full membership rounds with distinct endpoints: without pruning the
15091701 // retired set would double.
@@ -1512,8 +1704,8 @@ func TestRegistryPublishesBoundedTombstones(t *testing.T) {
15121704 for index := 0; index < ports.BrokerMaxHosts; index++ {
15131705 registrations = append(registrations, registration(t, fmt.Sprintf("round-%d-host-%d.test", round, index), byte(round*ports.BrokerMaxHosts+index+1)))
15141706 }
1515- require.NoError(t, registry.SetHosts(registrations))
1516- require.NoError(t, registry.SetHosts(nil))
1707+ require.NoError(t, registry.setHosts(registrations))
1708+ require.NoError(t, registry.setHosts(nil))
15171709 }
15181710
15191711 snapshot := registry.Snapshot()
@@ -1538,7 +1730,7 @@ func TestRegistryPersistenceRunsOffLockAndCoalescesWrites(t *testing.T) {
15381730 defer registry.store.close()
15391731
15401732 reg := registration(t, "example.test", 1)
1541- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
1733+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
15421734 // The writer is now blocked inside the first durable write.
15431735 first := <-store.entered
15441736 require.Equal(t, ports.BrokerRevision(2), first)
@@ -1551,7 +1743,7 @@ func TestRegistryPersistenceRunsOffLockAndCoalescesWrites(t *testing.T) {
15511743 defer close(published)
15521744 for generation := domain.RemoteGeneration(2); generation <= 6; generation++ {
15531745 reg.Generation = generation
1554- if err := registry.SetHosts([]domain.RemoteRegistration{reg}); err != nil {
1746+ if err := registry.setHosts([]domain.RemoteRegistration{reg}); err != nil {
15551747 setErr = err
15561748 return
15571749 }
@@ -1573,6 +1765,83 @@ func TestRegistryPersistenceRunsOffLockAndCoalescesWrites(t *testing.T) {
15731765 require.Equal(t, []ports.BrokerRevision{2, last}, store.writtenRevisions())
15741766 }
15751767
1768+// TestRegistryReplaceHostsBlockedCASKeepsSnapshotLockFree proves membership
1769+// durability and publication stay independent: a ReplaceHosts parked inside the
1770+// durable CAS may hold the registry lock, but Snapshot is served from the atomic
1771+// publication, no observation may be applied behind it, and the run loop
1772+// resumes scheduling as soon as the CAS commits.
1773+func TestRegistryReplaceHostsBlockedCASKeepsSnapshotLockFree(t *testing.T) {
1774+ clock := newManualClock(time.Unix(100, 0))
1775+ probe := newTestProbe(2)
1776+ store := newBlockingCASStore()
1777+ registry, err := NewRegistry(1, store, probe, clock, nil)
1778+ require.NoError(t, err)
1779+
1780+ reg := registration(t, "example.test", 1)
1781+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1782+ startRegistry(t, registry)
1783+ call := receiveCall(t, probe)
1784+ require.True(t, waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {
1785+ return host.Checking
1786+ }).Checking)
1787+
1788+ // The replacement takes the registry lock and parks inside the durable CAS.
1789+ replaced := make(chan error, 1)
1790+ go func() {
1791+ replaced <- registry.ReplaceHosts([]ports.BrokerHostRecord{{Registration: reg, Pinned: true, Policy: poolPolicy()}})
1792+ }()
1793+ <-store.entered
1794+ t.Cleanup(store.releaseCAS)
1795+
1796+ // Snapshot stays lock-free: the atomic publication is readable while the
1797+ // mutator is parked inside the store, and it still carries the pre-CAS
1798+ // projection.
1799+ observed := make(chan ports.BrokerSnapshot, 1)
1800+ go func() { observed <- registry.Snapshot() }()
1801+ select {
1802+ case snapshot := <-observed:
1803+ require.NoError(t, snapshot.Validate())
1804+ require.Len(t, snapshot.Hosts, 1)
1805+ require.Equal(t, reg, snapshot.Hosts[0].Registration)
1806+ case <-time.After(time.Second):
1807+ t.Fatal("Snapshot blocked behind the durable membership CAS")
1808+ }
1809+
1810+ // Wake the run loop and complete the observation while the CAS is still
1811+ // parked: applying the result needs the registry lock, so the in-flight
1812+ // projection must stay exactly as published until membership commits.
1813+ registry.hint()
1814+ call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{Availability: domain.RemoteAvailabilityReachable}}
1815+ pending, ok := registry.Snapshot().Find(reg.Endpoint)
1816+ require.True(t, ok)
1817+ require.True(t, pending.Checking)
1818+ stable := time.NewTimer(probeGrace)
1819+ defer stable.Stop()
1820+
1821+window:
1822+ for {
1823+ current, found := registry.Snapshot().Find(reg.Endpoint)
1824+ require.True(t, found)
1825+ require.Equal(t, pending, current)
1826+ select {
1827+ case <-stable.C:
1828+ break window
1829+ case <-time.After(time.Millisecond):
1830+ }
1831+ }
1832+
1833+ // Committing membership releases the mutator and unparks the run loop, which
1834+ // applies the observation and resumes scheduling the next one.
1835+ store.releaseCAS()
1836+ require.NoError(t, <-replaced)
1837+ host := waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {
1838+ return !host.Checking && host.Availability == domain.RemoteAvailabilityReachable
1839+ })
1840+ require.Equal(t, clock.Now(), host.LastSuccess)
1841+ clock.Advance(defaultFreshFor)
1842+ receiveCall(t, probe)
1843+}
1844+
15761845 func TestRegistryStoreErrorsAreObservableAndDoNotStallPublication(t *testing.T) {
15771846 clock := newManualClock(time.Unix(100, 0))
15781847 probe := newTestProbe(1)
@@ -1584,14 +1853,14 @@ func TestRegistryStoreErrorsAreObservableAndDoNotStallPublication(t *testing.T)
15841853 defer registry.store.close()
15851854
15861855 reg := registration(t, "example.test", 1)
1587- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
1856+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
15881857 waitForLog(t, logs, "durable snapshot write failed")
15891858 require.Contains(t, logs.String(), storeErr.Error())
15901859
15911860 // A failed write is not fatal and never blocks the next publication.
15921861 store.setStoreErr(nil)
15931862 reg.Generation++
1594- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
1863+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
15951864 stored := waitStored(t, store, func(snapshot ports.BrokerSnapshot) bool {
15961865 return snapshot.Revision == registry.Snapshot().Revision
15971866 })
@@ -1605,7 +1874,7 @@ func TestRegistryPersistedCopyClearsChecking(t *testing.T) {
16051874 store := &testStore{}
16061875 registry := newTestRegistry(t, 3, store, probe, clock)
16071876 reg := registration(t, "example.test", 1)
1608- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
1877+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
16091878 startRegistry(t, registry)
16101879
16111880 // The observation is in flight: the published projection carries Checking,
@@ -1627,16 +1896,48 @@ func TestRegistryPersistedCopyClearsChecking(t *testing.T) {
16271896 require.False(t, settled.Hosts[0].Checking)
16281897 }
16291898
1899+// TestRegistryNeverPersistsRetirementTombstones proves retirement tombstones
1900+// are process-local fencing state: the durable copy carries no Removed, and a
1901+// fresh registry reopened over that snapshot neither resurrects the retired
1902+// host nor sees a stale tombstone.
1903+func TestRegistryNeverPersistsRetirementTombstones(t *testing.T) {
1904+ clock := newManualClock(time.Unix(100, 0))
1905+ store := &testStore{}
1906+ registry := newTestRegistry(t, 1, store, newTestProbe(1), clock)
1907+ defer registry.store.close()
1908+
1909+ reg := registration(t, "retired.test", 1)
1910+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1911+ require.NoError(t, registry.setHosts(nil))
1912+
1913+ // Removal publishes a bounded tombstone and drops the live host in memory.
1914+ require.Empty(t, registry.Snapshot().Hosts)
1915+ require.Len(t, registry.Snapshot().Removed, 1)
1916+
1917+ // The durable copy drops the tombstone before the store ever sees it.
1918+ stored := waitStored(t, store, func(snapshot ports.BrokerSnapshot) bool {
1919+ return len(snapshot.Hosts) == 0 && snapshot.Revision == registry.Snapshot().Revision
1920+ })
1921+ require.Empty(t, stored.Removed)
1922+
1923+ // Reopening over the durable snapshot adopts empty membership without the
1924+ // stale tombstone.
1925+ reopened, err := NewRegistry(2, &testStore{loaded: stored}, newTestProbe(1), clock, nil)
1926+ require.NoError(t, err)
1927+ require.Empty(t, reopened.Snapshot().Hosts)
1928+ require.Empty(t, reopened.Snapshot().Removed)
1929+}
1930+
16301931 func TestRegistryBoundsSlowSubscribersAndStopsOnCancellation(t *testing.T) {
16311932 clock := newManualClock(time.Unix(100, 0))
16321933 probe := newTestProbe(1)
1633- registry := newTestRegistry(t, 1, nil, probe, clock)
1934+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
16341935
16351936 sub := registry.Subscribe()
16361937 reg := registration(t, "example.test", 1)
16371938 for generation := domain.RemoteGeneration(1); generation <= 10; generation++ {
16381939 reg.Generation = generation
1639- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
1940+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
16401941 }
16411942 // A subscriber that never drains is bounded to one pending wake and
16421943 // never blocks publication.
@@ -1648,7 +1949,7 @@ func TestRegistryBoundsSlowSubscribersAndStopsOnCancellation(t *testing.T) {
16481949 sub.Close()
16491950 sub.Close()
16501951 reg.Generation++
1651- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
1952+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
16521953 select {
16531954 case <-sub.Changed():
16541955 t.Fatal("closed subscription received a notification")
@@ -1685,7 +1986,7 @@ func TestRegistryRunIsSingleShotAndCleansUpInFlightState(t *testing.T) {
16851986 registry, err := NewRegistry(1, store, probe, clock, logger)
16861987 require.NoError(t, err)
16871988 reg := registration(t, "example.test", 1)
1688- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))
1989+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
16891990
16901991 ctx, cancel := context.WithCancel(context.Background())
16911992 done := make(chan struct{})
@@ -1730,7 +2031,7 @@ func TestRegistryRunIsSingleShotAndCleansUpInFlightState(t *testing.T) {
17302031 require.Equal(t, settled.Revision, stored.Revision)
17312032 require.False(t, stored.Hosts[0].Checking)
17322033 writes := store.writeCount()
1733- require.NoError(t, registry.SetHosts(nil))
2034+ require.NoError(t, registry.setHosts(nil))
17342035 require.Equal(t, writes, store.writeCount())
17352036 require.Contains(t, logs.String(), "refused after shutdown")
17362037
@@ -1749,7 +2050,7 @@ func TestRegistryRevisionsIncreaseMonotonicallyWithinEpoch(t *testing.T) {
17492050 start := time.Unix(100, 0)
17502051 clock := newManualClock(start)
17512052 probe := newTestProbe(2)
1752- registry := newTestRegistry(t, 42, nil, probe, clock)
2053+ registry := newTestRegistry(t, 42, newTestStore(), probe, clock)
17532054
17542055 var last ports.BrokerRevision
17552056 observe := func() {
@@ -1763,9 +2064,9 @@ func TestRegistryRevisionsIncreaseMonotonicallyWithinEpoch(t *testing.T) {
17632064
17642065 first := registration(t, "a.test", 1)
17652066 second := registration(t, "b.test", 2)
1766- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{first}))
2067+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{first}))
17672068 observe()
1768- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{first, second}))
2069+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{first, second}))
17692070 observe()
17702071
17712072 startRegistry(t, registry)
@@ -3,6 +3,7 @@ package broker
3 import (3 import (
4 "bytes"4 "bytes"
5 "context"5 "context"
6+ "crypto/sha256"
6 "errors"7 "errors"
7 "fmt"8 "fmt"
8 "log/slog"9 "log/slog"
@@ -165,11 +166,17 @@ func testLogger() (*slog.Logger, *syncBuffer) {
165 }166 }
166 167
167 // testStore is a race-safe in-memory durable store whose writes signal a168 // testStore is a race-safe in-memory durable store whose writes signal a
168-// capacity-one channel so tests wait on a write instead of on wall time.169+// capacity-one channel so tests wait on a write instead of on wall time. It
170+// implements the whole BrokerHostStore seam: authority defaults to the hosts of
171+// the restored snapshot, so a fixture that only names a durable snapshot still
172+// keeps exercising the rule that observations intersect authoritative
173+// membership.
169 type testStore struct {174 type testStore struct {
170 mu sync.Mutex175 mu sync.Mutex
171 loaded ports.BrokerSnapshot176 loaded ports.BrokerSnapshot
172 loadErr error177 loadErr error
178+ hosts ports.BrokerHosts
179+ hostsErr error
173 stored ports.BrokerSnapshot180 stored ports.BrokerSnapshot
174 storeErr error181 storeErr error
175 writes int182 writes int
@@ -185,6 +192,41 @@ func (s *testStore) Load() (ports.BrokerSnapshot, error) {
185 return s.loaded.Clone(), nil192 return s.loaded.Clone(), nil
186 }193 }
187 194
195+func (s *testStore) LoadHosts() (ports.BrokerHosts, error) {
196+ s.mu.Lock()
197+ defer s.mu.Unlock()
198+ if s.hostsErr != nil {
199+ return ports.BrokerHosts{}, s.hostsErr
200+ }
201+ if s.hosts.Revision != 0 || len(s.hosts.Hosts) > 0 {
202+ return s.hosts, nil
203+ }
204+ authority := ports.BrokerHosts{Revision: 1}
205+ for _, host := range s.loaded.Hosts {
206+ authority.Hosts = append(authority.Hosts, ports.BrokerHostRecord{Registration: host.Registration, Pinned: true, Policy: poolPolicy()})
207+ }
208+ return authority, nil
209+}
210+
211+func (s *testStore) ReplaceHosts(expected uint64, hosts []ports.BrokerHostRecord) error {
212+ s.mu.Lock()
213+ defer s.mu.Unlock()
214+ if s.hostsErr != nil {
215+ return s.hostsErr
216+ }
217+ if s.hosts.Revision == 0 {
218+ s.hosts.Revision = 1
219+ }
220+ if expected != s.hosts.Revision {
221+ return ports.ErrBrokerHostConflict
222+ }
223+ s.hosts.Hosts = append([]ports.BrokerHostRecord(nil), hosts...)
224+ s.hosts.Revision++
225+ return nil
226+}
227+
228+func (s *testStore) Close() error { return nil }
229+
188 func (s *testStore) Store(snapshot ports.BrokerSnapshot) error {230 func (s *testStore) Store(snapshot ports.BrokerSnapshot) error {
189 s.mu.Lock()231 s.mu.Lock()
190 s.stored = snapshot.Clone()232 s.stored = snapshot.Clone()
@@ -251,6 +293,14 @@ func newBlockingStore() *blockingStore {
251 293
252 func (s *blockingStore) Load() (ports.BrokerSnapshot, error) { return ports.BrokerSnapshot{}, nil }294 func (s *blockingStore) Load() (ports.BrokerSnapshot, error) { return ports.BrokerSnapshot{}, nil }
253 295
296+func (s *blockingStore) LoadHosts() (ports.BrokerHosts, error) {
297+ return ports.BrokerHosts{Revision: 1}, nil
298+}
299+
300+func (s *blockingStore) ReplaceHosts(uint64, []ports.BrokerHostRecord) error { return nil }
301+
302+func (s *blockingStore) Close() error { return nil }
303+
254 func (s *blockingStore) Store(snapshot ports.BrokerSnapshot) error {304 func (s *blockingStore) Store(snapshot ports.BrokerSnapshot) error {
255 s.mu.Lock()305 s.mu.Lock()
256 s.revisions = append(s.revisions, snapshot.Revision)306 s.revisions = append(s.revisions, snapshot.Revision)
@@ -273,6 +323,34 @@ func (s *blockingStore) writtenRevisions() []ports.BrokerRevision {
273 return append([]ports.BrokerRevision(nil), s.revisions...)323 return append([]ports.BrokerRevision(nil), s.revisions...)
274 }324 }
275 325
326+// blockingCASStore parks the first durable membership CAS until it is released,
327+// so a test can prove publication and Snapshot never wait behind membership
328+// durability. Load, LoadHosts, and Store keep the race-safe testStore behavior,
329+// and every later CAS is admitted without blocking.
330+type blockingCASStore struct {
331+ testStore
332+ entered chan struct{}
333+ released chan struct{}
334+ block sync.Once
335+ open sync.Once
336+}
337+
338+func newBlockingCASStore() *blockingCASStore {
339+ return &blockingCASStore{entered: make(chan struct{}, 1), released: make(chan struct{})}
340+}
341+
342+func (s *blockingCASStore) ReplaceHosts(expected uint64, hosts []ports.BrokerHostRecord) error {
343+ s.block.Do(func() {
344+ s.entered <- struct{}{}
345+ <-s.released
346+ })
347+ return s.testStore.ReplaceHosts(expected, hosts)
348+}
349+
350+// releaseCAS lets the parked CAS commit. It is idempotent so a failing test can
351+// unblock the parked mutator from cleanup.
352+func (s *blockingCASStore) releaseCAS() { s.open.Do(func() { close(s.released) }) }
353+
276 // testProbe scripts one observation per received call: the test owns the354 // testProbe scripts one observation per received call: the test owns the
277 // completion, so a probe never depends on wall time.355 // completion, so a probe never depends on wall time.
278 type testProbe struct {356 type testProbe struct {
@@ -421,22 +499,41 @@ func registration(t *testing.T, endpoint string, marker byte) domain.RemoteRegis
421 return result499 return result
422 }500 }
423 501
424-// hostSessions builds count minimal catalogue sessions for the projection502+// hostSession builds one catalogue-valid session. The registry applies the
503+// same durable projection rules as the store before adopting an observation, so
504+// a fixture session must carry a non-zero lifecycle identity and a non-nil tab
505+// list to count as durable.
506+func hostSession(name string) catalogue.RemoteCatalogSession {
507+ sum := sha256.Sum256([]byte("lifecycle:" + name))
508+ lifecycle := domain.SessionLifecycleID{}
509+ copy(lifecycle[:], sum[:])
510+ return catalogue.RemoteCatalogSession{
511+ LifecycleID: lifecycle,
512+ Name: name,
513+ State: catalogue.RemoteCatalogSessionUp,
514+ Tabs: []catalogue.RemoteCatalogTab{},
515+ }
516+}
517+
518+// hostSessions builds count minimal catalogue-valid sessions for the projection
425 // session bound tests.519 // session bound tests.
426 func hostSessions(count int) []catalogue.RemoteCatalogSession {520 func hostSessions(count int) []catalogue.RemoteCatalogSession {
427 sessions := make([]catalogue.RemoteCatalogSession, 0, count)521 sessions := make([]catalogue.RemoteCatalogSession, 0, count)
428 for index := 0; index < count; index++ {522 for index := 0; index < count; index++ {
429- sessions = append(sessions, catalogue.RemoteCatalogSession{523+ sessions = append(sessions, hostSession(fmt.Sprintf("session-%d", index)))
430- Name: fmt.Sprintf("session-%d", index),
431- State: catalogue.RemoteCatalogSessionUp,
432- })
433 }524 }
434 return sessions525 return sessions
435 }526 }
436 527
528+// newTestStore returns the minimal authoritative durable store: a valid
529+// revision series over empty membership. NewRegistry requires a
530+// BrokerHostStore, so every registry fixture names this store explicitly
531+// instead of relying on an optional seam.
532+func newTestStore() *testStore { return &testStore{} }
533+
437 // newTestRegistry builds a registry with exact scheduling and no logger so a534 // newTestRegistry builds a registry with exact scheduling and no logger so a
438 // failure never depends on jitter.535 // failure never depends on jitter.
439-func newTestRegistry(t *testing.T, epoch ports.BrokerEpoch, store ports.BrokerSnapshotStore, probe ports.BrokerHostProbe, clock *manualClock) *Registry {536+func newTestRegistry(t *testing.T, epoch ports.BrokerEpoch, store ports.BrokerHostStore, probe ports.BrokerHostProbe, clock *manualClock) *Registry {
440 t.Helper()537 t.Helper()
441 registry, err := NewRegistry(epoch, store, probe, clock, nil)538 registry, err := NewRegistry(epoch, store, probe, clock, nil)
442 require.NoError(t, err)539 require.NoError(t, err)
@@ -633,11 +730,19 @@ func TestRegistryRestoreRejectsInvalidInputs(t *testing.T) {
633 clock := newManualClock(time.Unix(100, 0))730 clock := newManualClock(time.Unix(100, 0))
634 reg := registration(t, "example.test", 1)731 reg := registration(t, "example.test", 1)
635 732
636- _, err := NewRegistry(0, nil, newTestProbe(1), clock, nil)733+ _, err := NewRegistry(0, newTestStore(), newTestProbe(1), clock, nil)
734+ require.Error(t, err)
735+ _, err = NewRegistry(1, newTestStore(), nil, clock, nil)
736+ require.Error(t, err)
737+ _, err = NewRegistry(1, newTestStore(), newTestProbe(1), nil, nil)
637 require.Error(t, err)738 require.Error(t, err)
638- _, err = NewRegistry(1, nil, nil, clock, nil)739+
740+ // The store is a required dependency: a literal nil store is rejected, and
741+ // a typed nil behind the interface is rejected without panicking.
742+ _, err = NewRegistry(1, nil, newTestProbe(1), clock, nil)
639 require.Error(t, err)743 require.Error(t, err)
640- _, err = NewRegistry(1, nil, newTestProbe(1), nil, nil)744+ var typedNilStore *testStore
745+ _, err = NewRegistry(1, typedNilStore, newTestProbe(1), clock, nil)
641 require.Error(t, err)746 require.Error(t, err)
642 747
643 loadErr := errors.New("load failed")748 loadErr := errors.New("load failed")
@@ -698,40 +803,40 @@ func TestRegistryRestoreRejectsPersistedSnapshotFromItsOwnEpoch(t *testing.T) {
698 }803 }
699 804
700 func TestRegistrySetHostsRejectsInvalidRegistrations(t *testing.T) {805 func TestRegistrySetHostsRejectsInvalidRegistrations(t *testing.T) {
701- registry := newTestRegistry(t, 1, nil, newTestProbe(1), newManualClock(time.Unix(100, 0)))806+ registry := newTestRegistry(t, 1, newTestStore(), newTestProbe(1), newManualClock(time.Unix(100, 0)))
702 807
703 reg := registration(t, "example.test", 1)808 reg := registration(t, "example.test", 1)
704- require.Error(t, registry.SetHosts([]domain.RemoteRegistration{reg, reg}))809+ require.Error(t, registry.setHosts([]domain.RemoteRegistration{reg, reg}))
705- require.Error(t, registry.SetHosts([]domain.RemoteRegistration{{Endpoint: reg.Endpoint}}))810+ require.Error(t, registry.setHosts([]domain.RemoteRegistration{{Endpoint: reg.Endpoint}}))
706 require.Empty(t, registry.Snapshot().Hosts)811 require.Empty(t, registry.Snapshot().Hosts)
707 }812 }
708 813
709 func TestRegistrySetHostsRejectsHostLimitWithoutMutation(t *testing.T) {814 func TestRegistrySetHostsRejectsHostLimitWithoutMutation(t *testing.T) {
710- registry := newTestRegistry(t, 1, nil, newTestProbe(1), newManualClock(time.Unix(100, 0)))815+ registry := newTestRegistry(t, 1, newTestStore(), newTestProbe(1), newManualClock(time.Unix(100, 0)))
711 reg := registration(t, "example.test", 1)816 reg := registration(t, "example.test", 1)
712- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))817+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
713 818
714 oversized := make([]domain.RemoteRegistration, 0, ports.BrokerMaxHosts+1)819 oversized := make([]domain.RemoteRegistration, 0, ports.BrokerMaxHosts+1)
715 for index := 0; index <= ports.BrokerMaxHosts; index++ {820 for index := 0; index <= ports.BrokerMaxHosts; index++ {
716 oversized = append(oversized, registration(t, fmt.Sprintf("host-%d.test", index), byte(index+1)))821 oversized = append(oversized, registration(t, fmt.Sprintf("host-%d.test", index), byte(index+1)))
717 }822 }
718 before := registry.Snapshot()823 before := registry.Snapshot()
719- require.Error(t, registry.SetHosts(oversized))824+ require.Error(t, registry.setHosts(oversized))
720 // The rejected call mutates nothing: same membership, same revision.825 // The rejected call mutates nothing: same membership, same revision.
721 require.Equal(t, before, registry.Snapshot())826 require.Equal(t, before, registry.Snapshot())
722 827
723 // The boundary itself is accepted.828 // The boundary itself is accepted.
724- require.NoError(t, registry.SetHosts(oversized[:ports.BrokerMaxHosts]))829+ require.NoError(t, registry.setHosts(oversized[:ports.BrokerMaxHosts]))
725 require.Len(t, registry.Snapshot().Hosts, ports.BrokerMaxHosts)830 require.Len(t, registry.Snapshot().Hosts, ports.BrokerMaxHosts)
726 }831 }
727 832
728 func TestRegistryUnchangedSetHostsIsNoOp(t *testing.T) {833 func TestRegistryUnchangedSetHostsIsNoOp(t *testing.T) {
729 clock := newManualClock(time.Unix(100, 0))834 clock := newManualClock(time.Unix(100, 0))
730 probe := newTestProbe(2)835 probe := newTestProbe(2)
731- registry := newTestRegistry(t, 1, nil, probe, clock)836+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
732 first := registration(t, "example.test", 1)837 first := registration(t, "example.test", 1)
733 second := registration(t, "other.test", 2)838 second := registration(t, "other.test", 2)
734- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{first, second}))839+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{first, second}))
735 startRegistry(t, registry)840 startRegistry(t, registry)
736 841
737 // Every configured host is observed exactly once.842 // Every configured host is observed exactly once.
@@ -748,8 +853,8 @@ func TestRegistryUnchangedSetHostsIsNoOp(t *testing.T) {
748 853
749 // Identical membership, including in a different order, never republishes854 // Identical membership, including in a different order, never republishes
750 // and never restarts an in-flight observation.855 // and never restarts an in-flight observation.
751- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{second, first}))856+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{second, first}))
752- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{first, second}))857+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{first, second}))
753 require.Equal(t, before, registry.Snapshot())858 require.Equal(t, before, registry.Snapshot())
754 requireNoProbeWithin(t, probe, probeGrace)859 requireNoProbeWithin(t, probe, probeGrace)
755 860
@@ -765,18 +870,19 @@ func TestRegistryUnchangedSetHostsIsNoOp(t *testing.T) {
765 func TestRegistrySnapshotIsImmutable(t *testing.T) {870 func TestRegistrySnapshotIsImmutable(t *testing.T) {
766 clock := newManualClock(time.Unix(100, 0))871 clock := newManualClock(time.Unix(100, 0))
767 probe := newTestProbe(1)872 probe := newTestProbe(1)
768- registry := newTestRegistry(t, 1, nil, probe, clock)873+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
769 reg := registration(t, "example.test", 1)874 reg := registration(t, "example.test", 1)
770- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))875+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
771 startRegistry(t, registry)876 startRegistry(t, registry)
772 877
773 call := receiveCall(t, probe)878 call := receiveCall(t, probe)
774 call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{879 call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{
775 Availability: domain.RemoteAvailabilityReachable,880 Availability: domain.RemoteAvailabilityReachable,
776 Sessions: []catalogue.RemoteCatalogSession{{881 Sessions: []catalogue.RemoteCatalogSession{{
777- Name: "work",882+ LifecycleID: domain.SessionLifecycleID{1},
778- State: catalogue.RemoteCatalogSessionUp,883+ Name: "work",
779- Tabs: []catalogue.RemoteCatalogTab{{ID: "tab-1", Name: "shell"}},884+ State: catalogue.RemoteCatalogSessionUp,
885+ Tabs: []catalogue.RemoteCatalogTab{{ID: "tab-1", Name: "shell"}},
780 }},886 }},
781 }}887 }}
782 observed := waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {888 observed := waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {
@@ -799,7 +905,7 @@ func TestRegistrySnapshotIsImmutable(t *testing.T) {
799 next := receiveCall(t, probe)905 next := receiveCall(t, probe)
800 next.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{906 next.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{
801 Availability: domain.RemoteAvailabilityReachable,907 Availability: domain.RemoteAvailabilityReachable,
802- Sessions: []catalogue.RemoteCatalogSession{{Name: "other", State: catalogue.RemoteCatalogSessionUp}},908+ Sessions: []catalogue.RemoteCatalogSession{hostSession("other")},
803 }}909 }}
804 waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {910 waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {
805 return !host.Checking && len(host.Sessions) == 1 && host.Sessions[0].Name == "other"911 return !host.Checking && len(host.Sessions) == 1 && host.Sessions[0].Name == "other"
@@ -811,9 +917,9 @@ func TestRegistrySchedulesFreshObservationAfterSuccess(t *testing.T) {
811 start := time.Unix(100, 0)917 start := time.Unix(100, 0)
812 clock := newManualClock(start)918 clock := newManualClock(start)
813 probe := newTestProbe(1)919 probe := newTestProbe(1)
814- registry := newTestRegistry(t, 1, nil, probe, clock)920+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
815 reg := registration(t, "example.test", 1)921 reg := registration(t, "example.test", 1)
816- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))922+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
817 startRegistry(t, registry)923 startRegistry(t, registry)
818 924
819 call := receiveCall(t, probe)925 call := receiveCall(t, probe)
@@ -840,9 +946,9 @@ func TestRegistryDemandOverridesFreshSchedule(t *testing.T) {
840 start := time.Unix(100, 0)946 start := time.Unix(100, 0)
841 clock := newManualClock(start)947 clock := newManualClock(start)
842 probe := newTestProbe(1)948 probe := newTestProbe(1)
843- registry := newTestRegistry(t, 1, nil, probe, clock)949+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
844 reg := registration(t, "example.test", 1)950 reg := registration(t, "example.test", 1)
845- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))951+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
846 startRegistry(t, registry)952 startRegistry(t, registry)
847 953
848 call := receiveCall(t, probe)954 call := receiveCall(t, probe)
@@ -862,9 +968,9 @@ func TestRegistryCoalescesConcurrentDemand(t *testing.T) {
862 start := time.Unix(100, 0)968 start := time.Unix(100, 0)
863 clock := newManualClock(start)969 clock := newManualClock(start)
864 probe := newTestProbe(2)970 probe := newTestProbe(2)
865- registry := newTestRegistry(t, 1, nil, probe, clock)971+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
866 reg := registration(t, "example.test", 1)972 reg := registration(t, "example.test", 1)
867- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))973+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
868 startRegistry(t, registry)974 startRegistry(t, registry)
869 975
870 call := receiveCall(t, probe)976 call := receiveCall(t, probe)
@@ -894,9 +1000,9 @@ func TestRegistryCoalescesConcurrentDemand(t *testing.T) {
894 func TestRegistryIgnoresDemandForUnknownEndpoint(t *testing.T) {1000 func TestRegistryIgnoresDemandForUnknownEndpoint(t *testing.T) {
895 clock := newManualClock(time.Unix(100, 0))1001 clock := newManualClock(time.Unix(100, 0))
896 probe := newTestProbe(1)1002 probe := newTestProbe(1)
897- registry := newTestRegistry(t, 1, nil, probe, clock)1003+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
898 reg := registration(t, "example.test", 1)1004 reg := registration(t, "example.test", 1)
899- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))1005+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
900 registry.RequestProbe("unknown.test")1006 registry.RequestProbe("unknown.test")
901 startRegistry(t, registry)1007 startRegistry(t, registry)
902 1008
@@ -913,14 +1019,14 @@ func TestRegistryPublishesReachabilityAndPersists(t *testing.T) {
913 store := &testStore{}1019 store := &testStore{}
914 registry := newTestRegistry(t, 3, store, probe, clock)1020 registry := newTestRegistry(t, 3, store, probe, clock)
915 reg := registration(t, "example.test", 1)1021 reg := registration(t, "example.test", 1)
916- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))1022+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
917 startRegistry(t, registry)1023 startRegistry(t, registry)
918 1024
919 call := receiveCall(t, probe)1025 call := receiveCall(t, probe)
920 call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{1026 call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{
921 Availability: domain.RemoteAvailabilityReachable,1027 Availability: domain.RemoteAvailabilityReachable,
922 InventoryKnown: true,1028 InventoryKnown: true,
923- Sessions: []catalogue.RemoteCatalogSession{{Name: "work", State: catalogue.RemoteCatalogSessionUp}},1029+ Sessions: []catalogue.RemoteCatalogSession{hostSession("work")},
924 }}1030 }}
925 snapshot := waitSnapshot(t, registry, func(snapshot ports.BrokerSnapshot) bool {1031 snapshot := waitSnapshot(t, registry, func(snapshot ports.BrokerSnapshot) bool {
926 host, ok := snapshot.Find(reg.Endpoint)1032 host, ok := snapshot.Find(reg.Endpoint)
@@ -996,9 +1102,9 @@ func TestRegistryNonReachableOutcomesUseTypedFailureBackoff(t *testing.T) {
996 start := time.Unix(100, 0)1102 start := time.Unix(100, 0)
997 clock := newManualClock(start)1103 clock := newManualClock(start)
998 probe := newTestProbe(1)1104 probe := newTestProbe(1)
999- registry := newTestRegistry(t, 1, nil, probe, clock)1105+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
1000 reg := registration(t, "example.test", 1)1106 reg := registration(t, "example.test", 1)
1001- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))1107+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1002 startRegistry(t, registry)1108 startRegistry(t, registry)
1003 1109
1004 call := receiveCall(t, probe)1110 call := receiveCall(t, probe)
@@ -1027,9 +1133,9 @@ func TestRegistryClassifiesOversizedProbeProjectionAsInvalidResponse(t *testing.
1027 start := time.Unix(100, 0)1133 start := time.Unix(100, 0)
1028 clock := newManualClock(start)1134 clock := newManualClock(start)
1029 probe := newTestProbe(1)1135 probe := newTestProbe(1)
1030- registry := newTestRegistry(t, 1, nil, probe, clock)1136+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
1031 reg := registration(t, "example.test", 1)1137 reg := registration(t, "example.test", 1)
1032- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))1138+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1033 startRegistry(t, registry)1139 startRegistry(t, registry)
1034 sub := registry.Subscribe()1140 sub := registry.Subscribe()
1035 defer sub.Close()1141 defer sub.Close()
@@ -1038,17 +1144,17 @@ func TestRegistryClassifiesOversizedProbeProjectionAsInvalidResponse(t *testing.
1038 call := receiveCall(t, probe)1144 call := receiveCall(t, probe)
1039 call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{1145 call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{
1040 Availability: domain.RemoteAvailabilityReachable,1146 Availability: domain.RemoteAvailabilityReachable,
1041- Sessions: []catalogue.RemoteCatalogSession{{Name: "work", State: catalogue.RemoteCatalogSessionUp}},1147+ Sessions: []catalogue.RemoteCatalogSession{hostSession("work")},
1042 }}1148 }}
1043 waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {1149 waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {
1044 return !host.Checking && host.Availability == domain.RemoteAvailabilityReachable1150 return !host.Checking && host.Availability == domain.RemoteAvailabilityReachable
1045 })1151 })
1046 1152
1047- // A projection carrying one session more than the publication bound is a1153+ // A projection carrying one session more than the durable session bound is
1048- // malformed response: it is classified as an invalid response and never1154+ // a malformed response: it is classified as an invalid response and never
1049- // reaches publication, because such a snapshot fails Validate. The loop1155+ // reaches publication, because the store enforces the same catalogue rule.
1050- // checks every observed publication, so an invalid snapshot can never pass1156+ // The loop checks every observed publication, so an invalid snapshot can
1051- // unnoticed.1157+ // never pass unnoticed.
1052 clock.Advance(defaultFreshFor)1158 clock.Advance(defaultFreshFor)
1053 call = receiveCall(t, probe)1159 call = receiveCall(t, probe)
1054 call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{1160 call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{
@@ -1084,9 +1190,9 @@ func TestRegistryAcceptsProbeProjectionAtSessionBound(t *testing.T) {
1084 start := time.Unix(100, 0)1190 start := time.Unix(100, 0)
1085 clock := newManualClock(start)1191 clock := newManualClock(start)
1086 probe := newTestProbe(1)1192 probe := newTestProbe(1)
1087- registry := newTestRegistry(t, 1, nil, probe, clock)1193+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
1088 reg := registration(t, "example.test", 1)1194 reg := registration(t, "example.test", 1)
1089- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))1195+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1090 startRegistry(t, registry)1196 startRegistry(t, registry)
1091 1197
1092 // The bound itself is valid: exactly BrokerMaxSessionsPerHost sessions are1198 // The bound itself is valid: exactly BrokerMaxSessionsPerHost sessions are
@@ -1106,20 +1212,106 @@ func TestRegistryAcceptsProbeProjectionAtSessionBound(t *testing.T) {
1106 require.Len(t, snapshot.Hosts[0].Sessions, ports.BrokerMaxSessionsPerHost)1212 require.Len(t, snapshot.Hosts[0].Sessions, ports.BrokerMaxSessionsPerHost)
1107 }1213 }
1108 1214
1215+// TestRegistryRejectsNonDurableObservation proves the registry and the durable
1216+// store agree on what may be persisted: a reachable projection the store would
1217+// reject is classified as an invalid response, so the last-known inventory
1218+// stays authoritative instead of a snapshot that can never be written.
1219+func TestRegistryRejectsNonDurableObservation(t *testing.T) {
1220+ start := time.Unix(100, 0)
1221+ clock := newManualClock(start)
1222+ probe := newTestProbe(1)
1223+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
1224+ reg := registration(t, "example.test", 1)
1225+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1226+ startRegistry(t, registry)
1227+
1228+ call := receiveCall(t, probe)
1229+ call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{
1230+ Availability: domain.RemoteAvailabilityReachable,
1231+ Sessions: []catalogue.RemoteCatalogSession{hostSession("work")},
1232+ }}
1233+ waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {
1234+ return !host.Checking && host.Availability == domain.RemoteAvailabilityReachable
1235+ })
1236+
1237+ // A probe answer that violates the catalogue rules the store enforces (here
1238+ // a zero lifecycle identity and an absent tab list) is a malformed response.
1239+ clock.Advance(defaultFreshFor)
1240+ call = receiveCall(t, probe)
1241+ call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{
1242+ Availability: domain.RemoteAvailabilityReachable,
1243+ Sessions: []catalogue.RemoteCatalogSession{{Name: "work", State: catalogue.RemoteCatalogSessionUp}},
1244+ }}
1245+ host := waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {
1246+ return !host.Checking && host.ConsecutiveFailures == 1
1247+ })
1248+ require.Equal(t, domain.RemoteAvailabilityInvalidResponse, host.Availability)
1249+ require.Equal(t, domain.RemoteFailureInvalidResponse, host.LastFailure.Kind)
1250+ require.Len(t, host.Sessions, 1)
1251+ require.Equal(t, "work", host.Sessions[0].Name)
1252+ require.Equal(t, start, host.LastSuccess)
1253+ snapshot := registry.Snapshot()
1254+ require.NoError(t, snapshot.Validate())
1255+ require.NoError(t, ports.ValidateDurableHostProjection(snapshot.Hosts[0]))
1256+}
1257+
1258+// TestRegistryRevisionOverflowFailsClosed pins the revision series: zero is not
1259+// a valid revision and revisions never regress, so an exhausted counter fails
1260+// closed instead of wrapping. The last valid snapshot stays published, the
1261+// registry is poisoned, and every mutation path reports a typed error rather
1262+// than emitting revision zero or a revision below the newest one, either of
1263+// which would silently freeze durable persistence.
1264+func TestRegistryRevisionOverflowFailsClosed(t *testing.T) {
1265+ logger, logs := testLogger()
1266+ registry, err := NewRegistry(1, newTestStore(), newTestProbe(1), newManualClock(time.Unix(100, 0)), logger)
1267+ require.NoError(t, err)
1268+ first := registration(t, "example.test", 1)
1269+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{first}))
1270+ before := registry.Snapshot()
1271+ require.NoError(t, before.Validate())
1272+ require.Equal(t, ports.BrokerRevision(2), before.Revision)
1273+ authority := registry.authority
1274+
1275+ // The next publication would overflow the series to zero.
1276+ registry.revision = ^ports.BrokerRevision(0)
1277+
1278+ // Both mutation paths refuse the change before touching the store or the
1279+ // projection, so neither durable authority nor the publication series moves.
1280+ require.ErrorIs(t, registry.setHosts([]domain.RemoteRegistration{registration(t, "other.test", 2)}), ports.ErrBrokerRevisionExhausted)
1281+ require.ErrorIs(t, registry.ReplaceHosts([]ports.BrokerHostRecord{{Registration: first, Pinned: true, Policy: poolPolicy()}}), ports.ErrBrokerRevisionExhausted)
1282+ require.Equal(t, authority, registry.authority)
1283+
1284+ // The publication path itself fails closed too: a direct attempt neither
1285+ // wraps the counter nor replaces the last valid snapshot.
1286+ registry.mu.Lock()
1287+ registry.publishLocked(false)
1288+ registry.mu.Unlock()
1289+
1290+ after := registry.Snapshot()
1291+ require.Equal(t, before, after)
1292+ require.NoError(t, after.Validate())
1293+ require.Greater(t, after.Revision, ports.BrokerRevision(0))
1294+ require.Equal(t, ^ports.BrokerRevision(0), registry.revision)
1295+ require.Equal(t, first, after.Hosts[0].Registration)
1296+ // The refusal is observable: an exhausted series logs instead of failing
1297+ // silently behind a frozen snapshot.
1298+ require.Contains(t, logs.String(), "revision series exhausted")
1299+}
1300+
1109 func TestRegistryPreservesLastSuccessAndOwnsFailureEpisode(t *testing.T) {1301 func TestRegistryPreservesLastSuccessAndOwnsFailureEpisode(t *testing.T) {
1110 start := time.Unix(100, 0)1302 start := time.Unix(100, 0)
1111 clock := newManualClock(start)1303 clock := newManualClock(start)
1112 probe := newTestProbe(1)1304 probe := newTestProbe(1)
1113- registry := newTestRegistry(t, 1, nil, probe, clock)1305+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
1114 reg := registration(t, "example.test", 1)1306 reg := registration(t, "example.test", 1)
1115- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))1307+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1116 startRegistry(t, registry)1308 startRegistry(t, registry)
1117 1309
1118 // A successful observation owns the success time and opens no episode.1310 // A successful observation owns the success time and opens no episode.
1119 call := receiveCall(t, probe)1311 call := receiveCall(t, probe)
1120 call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{1312 call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{
1121 Availability: domain.RemoteAvailabilityReachable,1313 Availability: domain.RemoteAvailabilityReachable,
1122- Sessions: []catalogue.RemoteCatalogSession{{Name: "work", State: catalogue.RemoteCatalogSessionUp}},1314+ Sessions: []catalogue.RemoteCatalogSession{hostSession("work")},
1123 }}1315 }}
1124 host := waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {1316 host := waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {
1125 return !host.Checking && host.Availability == domain.RemoteAvailabilityReachable1317 return !host.Checking && host.Availability == domain.RemoteAvailabilityReachable
@@ -1172,9 +1364,9 @@ func TestRegistryBackoffIsExponentialAndCapped(t *testing.T) {
1172 start := time.Unix(100, 0)1364 start := time.Unix(100, 0)
1173 clock := newManualClock(start)1365 clock := newManualClock(start)
1174 probe := newTestProbe(1)1366 probe := newTestProbe(1)
1175- registry := newTestRegistry(t, 1, nil, probe, clock)1367+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
1176 reg := registration(t, "example.test", 1)1368 reg := registration(t, "example.test", 1)
1177- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))1369+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1178 startRegistry(t, registry)1370 startRegistry(t, registry)
1179 1371
1180 probeErr := errors.New("offline")1372 probeErr := errors.New("offline")
@@ -1215,10 +1407,10 @@ func TestRegistryJitterAppliesToRetryAndHealthyRefresh(t *testing.T) {
1215 start := time.Unix(100, 0)1407 start := time.Unix(100, 0)
1216 clock := newManualClock(start)1408 clock := newManualClock(start)
1217 probe := newTestProbe(1)1409 probe := newTestProbe(1)
1218- registry := newTestRegistry(t, 1, nil, probe, clock)1410+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
1219 registry.jitter = func(base time.Duration, _ string, _ uint64) time.Duration { return base / 2 }1411 registry.jitter = func(base time.Duration, _ string, _ uint64) time.Duration { return base / 2 }
1220 reg := registration(t, "example.test", 1)1412 reg := registration(t, "example.test", 1)
1221- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))1413+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1222 startRegistry(t, registry)1414 startRegistry(t, registry)
1223 1415
1224 call := receiveCall(t, probe)1416 call := receiveCall(t, probe)
@@ -1267,20 +1459,20 @@ func TestRegistryFencesStaleCompletionAcrossRegistrationChanges(t *testing.T) {
1267 replacementAnswer := func(name string) probeAnswer {1459 replacementAnswer := func(name string) probeAnswer {
1268 return probeAnswer{snapshot: ports.RemoteHostSnapshot{1460 return probeAnswer{snapshot: ports.RemoteHostSnapshot{
1269 Availability: domain.RemoteAvailabilityReachable,1461 Availability: domain.RemoteAvailabilityReachable,
1270- Sessions: []catalogue.RemoteCatalogSession{{Name: name, State: catalogue.RemoteCatalogSessionUp}},1462+ Sessions: []catalogue.RemoteCatalogSession{hostSession(name)},
1271 }}1463 }}
1272 }1464 }
1273 assertFenced := func(t *testing.T, current, replacement domain.RemoteRegistration) {1465 assertFenced := func(t *testing.T, current, replacement domain.RemoteRegistration) {
1274 t.Helper()1466 t.Helper()
1275 clock := newManualClock(time.Unix(100, 0))1467 clock := newManualClock(time.Unix(100, 0))
1276 probe := newTestProbe(2)1468 probe := newTestProbe(2)
1277- registry := newTestRegistry(t, 1, nil, probe, clock)1469+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
1278- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{current}))1470+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{current}))
1279 startRegistry(t, registry)1471 startRegistry(t, registry)
1280 currentCall := receiveCall(t, probe)1472 currentCall := receiveCall(t, probe)
1281 require.True(t, currentCall.registration.Equal(current))1473 require.True(t, currentCall.registration.Equal(current))
1282 1474
1283- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{replacement}))1475+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{replacement}))
1284 replacementCall := receiveCall(t, probe)1476 replacementCall := receiveCall(t, probe)
1285 require.True(t, replacementCall.registration.Equal(replacement))1477 require.True(t, replacementCall.registration.Equal(replacement))
1286 1478
@@ -1322,9 +1514,9 @@ func TestRegistryFencesStaleCompletionAcrossRegistrationChanges(t *testing.T) {
1322 func TestRegistryFencesRetiredAttemptsAcrossRemovalAndIdenticalReAdd(t *testing.T) {1514 func TestRegistryFencesRetiredAttemptsAcrossRemovalAndIdenticalReAdd(t *testing.T) {
1323 clock := newManualClock(time.Unix(100, 0))1515 clock := newManualClock(time.Unix(100, 0))
1324 probe := newTestProbe(2)1516 probe := newTestProbe(2)
1325- registry := newTestRegistry(t, 1, nil, probe, clock)1517+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
1326 reg := registration(t, "example.test", 1)1518 reg := registration(t, "example.test", 1)
1327- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))1519+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1328 1520
1329 ctx, cancel := context.WithCancel(context.Background())1521 ctx, cancel := context.WithCancel(context.Background())
1330 defer cancel()1522 defer cancel()
@@ -1338,7 +1530,7 @@ func TestRegistryFencesRetiredAttemptsAcrossRemovalAndIdenticalReAdd(t *testing.
1338 require.NotZero(t, retired)1530 require.NotZero(t, retired)
1339 1531
1340 // Removal retires a bounded tombstone and drops the in-flight token.1532 // Removal retires a bounded tombstone and drops the in-flight token.
1341- require.NoError(t, registry.SetHosts(nil))1533+ require.NoError(t, registry.setHosts(nil))
1342 removed := registry.Snapshot()1534 removed := registry.Snapshot()
1343 require.Empty(t, removed.Hosts)1535 require.Empty(t, removed.Hosts)
1344 require.Len(t, removed.Removed, 1)1536 require.Len(t, removed.Removed, 1)
@@ -1353,7 +1545,7 @@ func TestRegistryFencesRetiredAttemptsAcrossRemovalAndIdenticalReAdd(t *testing.
1353 1545
1354 // Re-adding the identical registration supersedes the tombstone, so only1546 // Re-adding the identical registration supersedes the tombstone, so only
1355 // the attempt token can fence the retired completion.1547 // the attempt token can fence the retired completion.
1356- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))1548+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1357 require.Empty(t, registry.Snapshot().Removed)1549 require.Empty(t, registry.Snapshot().Removed)
1358 dispatch()1550 dispatch()
1359 freshCall := receiveCall(t, probe)1551 freshCall := receiveCall(t, probe)
@@ -1388,8 +1580,8 @@ func TestRegistryRetiresInflightAttemptsOnReplacementAndRemoval(t *testing.T) {
1388 t.Run(tt.name, func(t *testing.T) {1580 t.Run(tt.name, func(t *testing.T) {
1389 clock := newManualClock(time.Unix(100, 0))1581 clock := newManualClock(time.Unix(100, 0))
1390 probe := newTestProbe(2)1582 probe := newTestProbe(2)
1391- registry := newTestRegistry(t, 1, nil, probe, clock)1583+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
1392- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{current}))1584+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{current}))
1393 startRegistry(t, registry)1585 startRegistry(t, registry)
1394 1586
1395 call := receiveCall(t, probe)1587 call := receiveCall(t, probe)
@@ -1399,7 +1591,7 @@ func TestRegistryRetiresInflightAttemptsOnReplacementAndRemoval(t *testing.T) {
1399 // attempt: the probe's derived context is cancelled before the1591 // attempt: the probe's derived context is cancelled before the
1400 // replacement can start, so a ctx-honoring probe stops instead of1592 // replacement can start, so a ctx-honoring probe stops instead of
1401 // overlapping its successor.1593 // overlapping its successor.
1402- require.NoError(t, registry.SetHosts(tt.next))1594+ require.NoError(t, registry.setHosts(tt.next))
1403 select {1595 select {
1404 case <-probe.canceled:1596 case <-probe.canceled:
1405 case <-time.After(time.Second):1597 case <-time.After(time.Second):
@@ -1413,11 +1605,11 @@ func TestRegistryChurnKeepsOneActiveProbePerEndpoint(t *testing.T) {
1413 clock := newManualClock(time.Unix(100, 0))1605 clock := newManualClock(time.Unix(100, 0))
1414 probe := newChurnProbe()1606 probe := newChurnProbe()
1415 defer probe.openRelease()1607 defer probe.openRelease()
1416- registry := newTestRegistry(t, 1, nil, probe, clock)1608+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
1417 1609
1418 first := registration(t, "example.test", 1)1610 first := registration(t, "example.test", 1)
1419 second := registration(t, "example.test", 2)1611 second := registration(t, "example.test", 2)
1420- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{first}))1612+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{first}))
1421 1613
1422 ctx, cancel := context.WithCancel(context.Background())1614 ctx, cancel := context.WithCancel(context.Background())
1423 defer cancel()1615 defer cancel()
@@ -1443,7 +1635,7 @@ func TestRegistryChurnKeepsOneActiveProbePerEndpoint(t *testing.T) {
1443 if round%2 == 0 {1635 if round%2 == 0 {
1444 next = second1636 next = second
1445 }1637 }
1446- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{next}))1638+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{next}))
1447 select {1639 select {
1448 case <-probe.startedSig:1640 case <-probe.startedSig:
1449 case <-time.After(time.Second):1641 case <-time.After(time.Second):
@@ -1465,7 +1657,7 @@ func TestRegistryChurnKeepsOneActiveProbePerEndpoint(t *testing.T) {
1465 if (worker+round)%2 == 0 {1657 if (worker+round)%2 == 0 {
1466 next = second1658 next = second
1467 }1659 }
1468- if err := registry.SetHosts([]domain.RemoteRegistration{next}); err != nil {1660+ if err := registry.setHosts([]domain.RemoteRegistration{next}); err != nil {
1469 failures[worker] = err1661 failures[worker] = err
1470 return1662 return
1471 }1663 }
@@ -1503,7 +1695,7 @@ func TestRegistryChurnKeepsOneActiveProbePerEndpoint(t *testing.T) {
1503 }1695 }
1504 1696
1505 func TestRegistryPublishesBoundedTombstones(t *testing.T) {1697 func TestRegistryPublishesBoundedTombstones(t *testing.T) {
1506- registry := newTestRegistry(t, 1, nil, newTestProbe(1), newManualClock(time.Unix(100, 0)))1698+ registry := newTestRegistry(t, 1, newTestStore(), newTestProbe(1), newManualClock(time.Unix(100, 0)))
1507 1699
1508 // Two full membership rounds with distinct endpoints: without pruning the1700 // Two full membership rounds with distinct endpoints: without pruning the
1509 // retired set would double.1701 // retired set would double.
@@ -1512,8 +1704,8 @@ func TestRegistryPublishesBoundedTombstones(t *testing.T) {
1512 for index := 0; index < ports.BrokerMaxHosts; index++ {1704 for index := 0; index < ports.BrokerMaxHosts; index++ {
1513 registrations = append(registrations, registration(t, fmt.Sprintf("round-%d-host-%d.test", round, index), byte(round*ports.BrokerMaxHosts+index+1)))1705 registrations = append(registrations, registration(t, fmt.Sprintf("round-%d-host-%d.test", round, index), byte(round*ports.BrokerMaxHosts+index+1)))
1514 }1706 }
1515- require.NoError(t, registry.SetHosts(registrations))1707+ require.NoError(t, registry.setHosts(registrations))
1516- require.NoError(t, registry.SetHosts(nil))1708+ require.NoError(t, registry.setHosts(nil))
1517 }1709 }
1518 1710
1519 snapshot := registry.Snapshot()1711 snapshot := registry.Snapshot()
@@ -1538,7 +1730,7 @@ func TestRegistryPersistenceRunsOffLockAndCoalescesWrites(t *testing.T) {
1538 defer registry.store.close()1730 defer registry.store.close()
1539 1731
1540 reg := registration(t, "example.test", 1)1732 reg := registration(t, "example.test", 1)
1541- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))1733+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1542 // The writer is now blocked inside the first durable write.1734 // The writer is now blocked inside the first durable write.
1543 first := <-store.entered1735 first := <-store.entered
1544 require.Equal(t, ports.BrokerRevision(2), first)1736 require.Equal(t, ports.BrokerRevision(2), first)
@@ -1551,7 +1743,7 @@ func TestRegistryPersistenceRunsOffLockAndCoalescesWrites(t *testing.T) {
1551 defer close(published)1743 defer close(published)
1552 for generation := domain.RemoteGeneration(2); generation <= 6; generation++ {1744 for generation := domain.RemoteGeneration(2); generation <= 6; generation++ {
1553 reg.Generation = generation1745 reg.Generation = generation
1554- if err := registry.SetHosts([]domain.RemoteRegistration{reg}); err != nil {1746+ if err := registry.setHosts([]domain.RemoteRegistration{reg}); err != nil {
1555 setErr = err1747 setErr = err
1556 return1748 return
1557 }1749 }
@@ -1573,6 +1765,83 @@ func TestRegistryPersistenceRunsOffLockAndCoalescesWrites(t *testing.T) {
1573 require.Equal(t, []ports.BrokerRevision{2, last}, store.writtenRevisions())1765 require.Equal(t, []ports.BrokerRevision{2, last}, store.writtenRevisions())
1574 }1766 }
1575 1767
1768+// TestRegistryReplaceHostsBlockedCASKeepsSnapshotLockFree proves membership
1769+// durability and publication stay independent: a ReplaceHosts parked inside the
1770+// durable CAS may hold the registry lock, but Snapshot is served from the atomic
1771+// publication, no observation may be applied behind it, and the run loop
1772+// resumes scheduling as soon as the CAS commits.
1773+func TestRegistryReplaceHostsBlockedCASKeepsSnapshotLockFree(t *testing.T) {
1774+ clock := newManualClock(time.Unix(100, 0))
1775+ probe := newTestProbe(2)
1776+ store := newBlockingCASStore()
1777+ registry, err := NewRegistry(1, store, probe, clock, nil)
1778+ require.NoError(t, err)
1779+
1780+ reg := registration(t, "example.test", 1)
1781+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1782+ startRegistry(t, registry)
1783+ call := receiveCall(t, probe)
1784+ require.True(t, waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {
1785+ return host.Checking
1786+ }).Checking)
1787+
1788+ // The replacement takes the registry lock and parks inside the durable CAS.
1789+ replaced := make(chan error, 1)
1790+ go func() {
1791+ replaced <- registry.ReplaceHosts([]ports.BrokerHostRecord{{Registration: reg, Pinned: true, Policy: poolPolicy()}})
1792+ }()
1793+ <-store.entered
1794+ t.Cleanup(store.releaseCAS)
1795+
1796+ // Snapshot stays lock-free: the atomic publication is readable while the
1797+ // mutator is parked inside the store, and it still carries the pre-CAS
1798+ // projection.
1799+ observed := make(chan ports.BrokerSnapshot, 1)
1800+ go func() { observed <- registry.Snapshot() }()
1801+ select {
1802+ case snapshot := <-observed:
1803+ require.NoError(t, snapshot.Validate())
1804+ require.Len(t, snapshot.Hosts, 1)
1805+ require.Equal(t, reg, snapshot.Hosts[0].Registration)
1806+ case <-time.After(time.Second):
1807+ t.Fatal("Snapshot blocked behind the durable membership CAS")
1808+ }
1809+
1810+ // Wake the run loop and complete the observation while the CAS is still
1811+ // parked: applying the result needs the registry lock, so the in-flight
1812+ // projection must stay exactly as published until membership commits.
1813+ registry.hint()
1814+ call.result <- probeAnswer{snapshot: ports.RemoteHostSnapshot{Availability: domain.RemoteAvailabilityReachable}}
1815+ pending, ok := registry.Snapshot().Find(reg.Endpoint)
1816+ require.True(t, ok)
1817+ require.True(t, pending.Checking)
1818+ stable := time.NewTimer(probeGrace)
1819+ defer stable.Stop()
1820+
1821+window:
1822+ for {
1823+ current, found := registry.Snapshot().Find(reg.Endpoint)
1824+ require.True(t, found)
1825+ require.Equal(t, pending, current)
1826+ select {
1827+ case <-stable.C:
1828+ break window
1829+ case <-time.After(time.Millisecond):
1830+ }
1831+ }
1832+
1833+ // Committing membership releases the mutator and unparks the run loop, which
1834+ // applies the observation and resumes scheduling the next one.
1835+ store.releaseCAS()
1836+ require.NoError(t, <-replaced)
1837+ host := waitHost(t, registry, reg.Endpoint, func(host ports.RemoteHostSnapshot) bool {
1838+ return !host.Checking && host.Availability == domain.RemoteAvailabilityReachable
1839+ })
1840+ require.Equal(t, clock.Now(), host.LastSuccess)
1841+ clock.Advance(defaultFreshFor)
1842+ receiveCall(t, probe)
1843+}
1844+
1576 func TestRegistryStoreErrorsAreObservableAndDoNotStallPublication(t *testing.T) {1845 func TestRegistryStoreErrorsAreObservableAndDoNotStallPublication(t *testing.T) {
1577 clock := newManualClock(time.Unix(100, 0))1846 clock := newManualClock(time.Unix(100, 0))
1578 probe := newTestProbe(1)1847 probe := newTestProbe(1)
@@ -1584,14 +1853,14 @@ func TestRegistryStoreErrorsAreObservableAndDoNotStallPublication(t *testing.T)
1584 defer registry.store.close()1853 defer registry.store.close()
1585 1854
1586 reg := registration(t, "example.test", 1)1855 reg := registration(t, "example.test", 1)
1587- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))1856+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1588 waitForLog(t, logs, "durable snapshot write failed")1857 waitForLog(t, logs, "durable snapshot write failed")
1589 require.Contains(t, logs.String(), storeErr.Error())1858 require.Contains(t, logs.String(), storeErr.Error())
1590 1859
1591 // A failed write is not fatal and never blocks the next publication.1860 // A failed write is not fatal and never blocks the next publication.
1592 store.setStoreErr(nil)1861 store.setStoreErr(nil)
1593 reg.Generation++1862 reg.Generation++
1594- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))1863+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1595 stored := waitStored(t, store, func(snapshot ports.BrokerSnapshot) bool {1864 stored := waitStored(t, store, func(snapshot ports.BrokerSnapshot) bool {
1596 return snapshot.Revision == registry.Snapshot().Revision1865 return snapshot.Revision == registry.Snapshot().Revision
1597 })1866 })
@@ -1605,7 +1874,7 @@ func TestRegistryPersistedCopyClearsChecking(t *testing.T) {
1605 store := &testStore{}1874 store := &testStore{}
1606 registry := newTestRegistry(t, 3, store, probe, clock)1875 registry := newTestRegistry(t, 3, store, probe, clock)
1607 reg := registration(t, "example.test", 1)1876 reg := registration(t, "example.test", 1)
1608- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))1877+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1609 startRegistry(t, registry)1878 startRegistry(t, registry)
1610 1879
1611 // The observation is in flight: the published projection carries Checking,1880 // The observation is in flight: the published projection carries Checking,
@@ -1627,16 +1896,48 @@ func TestRegistryPersistedCopyClearsChecking(t *testing.T) {
1627 require.False(t, settled.Hosts[0].Checking)1896 require.False(t, settled.Hosts[0].Checking)
1628 }1897 }
1629 1898
1899+// TestRegistryNeverPersistsRetirementTombstones proves retirement tombstones
1900+// are process-local fencing state: the durable copy carries no Removed, and a
1901+// fresh registry reopened over that snapshot neither resurrects the retired
1902+// host nor sees a stale tombstone.
1903+func TestRegistryNeverPersistsRetirementTombstones(t *testing.T) {
1904+ clock := newManualClock(time.Unix(100, 0))
1905+ store := &testStore{}
1906+ registry := newTestRegistry(t, 1, store, newTestProbe(1), clock)
1907+ defer registry.store.close()
1908+
1909+ reg := registration(t, "retired.test", 1)
1910+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1911+ require.NoError(t, registry.setHosts(nil))
1912+
1913+ // Removal publishes a bounded tombstone and drops the live host in memory.
1914+ require.Empty(t, registry.Snapshot().Hosts)
1915+ require.Len(t, registry.Snapshot().Removed, 1)
1916+
1917+ // The durable copy drops the tombstone before the store ever sees it.
1918+ stored := waitStored(t, store, func(snapshot ports.BrokerSnapshot) bool {
1919+ return len(snapshot.Hosts) == 0 && snapshot.Revision == registry.Snapshot().Revision
1920+ })
1921+ require.Empty(t, stored.Removed)
1922+
1923+ // Reopening over the durable snapshot adopts empty membership without the
1924+ // stale tombstone.
1925+ reopened, err := NewRegistry(2, &testStore{loaded: stored}, newTestProbe(1), clock, nil)
1926+ require.NoError(t, err)
1927+ require.Empty(t, reopened.Snapshot().Hosts)
1928+ require.Empty(t, reopened.Snapshot().Removed)
1929+}
1930+
1630 func TestRegistryBoundsSlowSubscribersAndStopsOnCancellation(t *testing.T) {1931 func TestRegistryBoundsSlowSubscribersAndStopsOnCancellation(t *testing.T) {
1631 clock := newManualClock(time.Unix(100, 0))1932 clock := newManualClock(time.Unix(100, 0))
1632 probe := newTestProbe(1)1933 probe := newTestProbe(1)
1633- registry := newTestRegistry(t, 1, nil, probe, clock)1934+ registry := newTestRegistry(t, 1, newTestStore(), probe, clock)
1634 1935
1635 sub := registry.Subscribe()1936 sub := registry.Subscribe()
1636 reg := registration(t, "example.test", 1)1937 reg := registration(t, "example.test", 1)
1637 for generation := domain.RemoteGeneration(1); generation <= 10; generation++ {1938 for generation := domain.RemoteGeneration(1); generation <= 10; generation++ {
1638 reg.Generation = generation1939 reg.Generation = generation
1639- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))1940+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1640 }1941 }
1641 // A subscriber that never drains is bounded to one pending wake and1942 // A subscriber that never drains is bounded to one pending wake and
1642 // never blocks publication.1943 // never blocks publication.
@@ -1648,7 +1949,7 @@ func TestRegistryBoundsSlowSubscribersAndStopsOnCancellation(t *testing.T) {
1648 sub.Close()1949 sub.Close()
1649 sub.Close()1950 sub.Close()
1650 reg.Generation++1951 reg.Generation++
1651- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))1952+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1652 select {1953 select {
1653 case <-sub.Changed():1954 case <-sub.Changed():
1654 t.Fatal("closed subscription received a notification")1955 t.Fatal("closed subscription received a notification")
@@ -1685,7 +1986,7 @@ func TestRegistryRunIsSingleShotAndCleansUpInFlightState(t *testing.T) {
1685 registry, err := NewRegistry(1, store, probe, clock, logger)1986 registry, err := NewRegistry(1, store, probe, clock, logger)
1686 require.NoError(t, err)1987 require.NoError(t, err)
1687 reg := registration(t, "example.test", 1)1988 reg := registration(t, "example.test", 1)
1688- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{reg}))1989+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{reg}))
1689 1990
1690 ctx, cancel := context.WithCancel(context.Background())1991 ctx, cancel := context.WithCancel(context.Background())
1691 done := make(chan struct{})1992 done := make(chan struct{})
@@ -1730,7 +2031,7 @@ func TestRegistryRunIsSingleShotAndCleansUpInFlightState(t *testing.T) {
1730 require.Equal(t, settled.Revision, stored.Revision)2031 require.Equal(t, settled.Revision, stored.Revision)
1731 require.False(t, stored.Hosts[0].Checking)2032 require.False(t, stored.Hosts[0].Checking)
1732 writes := store.writeCount()2033 writes := store.writeCount()
1733- require.NoError(t, registry.SetHosts(nil))2034+ require.NoError(t, registry.setHosts(nil))
1734 require.Equal(t, writes, store.writeCount())2035 require.Equal(t, writes, store.writeCount())
1735 require.Contains(t, logs.String(), "refused after shutdown")2036 require.Contains(t, logs.String(), "refused after shutdown")
1736 2037
@@ -1749,7 +2050,7 @@ func TestRegistryRevisionsIncreaseMonotonicallyWithinEpoch(t *testing.T) {
1749 start := time.Unix(100, 0)2050 start := time.Unix(100, 0)
1750 clock := newManualClock(start)2051 clock := newManualClock(start)
1751 probe := newTestProbe(2)2052 probe := newTestProbe(2)
1752- registry := newTestRegistry(t, 42, nil, probe, clock)2053+ registry := newTestRegistry(t, 42, newTestStore(), probe, clock)
1753 2054
1754 var last ports.BrokerRevision2055 var last ports.BrokerRevision
1755 observe := func() {2056 observe := func() {
@@ -1763,9 +2064,9 @@ func TestRegistryRevisionsIncreaseMonotonicallyWithinEpoch(t *testing.T) {
1763 2064
1764 first := registration(t, "a.test", 1)2065 first := registration(t, "a.test", 1)
1765 second := registration(t, "b.test", 2)2066 second := registration(t, "b.test", 2)
1766- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{first}))2067+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{first}))
1767 observe()2068 observe()
1768- require.NoError(t, registry.SetHosts([]domain.RemoteRegistration{first, second}))2069+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{first, second}))
1769 observe()2070 observe()
1770 2071
1771 startRegistry(t, registry)2072 startRegistry(t, registry)

Keyboard shortcuts

Single-key shortcuts are on. ctrl k and esc work whatever this says.

ctrl k :
go to a section, branch, tag, file or repository
?
this list
j k
move the row cursor down / up
enter
open the row under the cursor; on Files, a directory opens or closes in place
↑ ↓ ← → home end
in the file tree (focus on a row): previous / next row, close / open a directory or go to its parent / first entry, first / last row. They never open a file
l h
on Files: open a directory or a file / close a directory or go up to its parent, then to the parent page. Elsewhere: into the side panel (on a merge request, its files) / back to the content; there j k move between its controls
g g G
first / last row
/
filter the list; on Files, find a file anywhere in the repository (esc clears, enter opens the best match)
b
switch branch or tag
g s g f g c g b g m g i g r g a
tabs: summary, files, commits, branches & tags, merge requests, issues, releases, actions (when available)
g h
home
w
wide mode: use the whole width of the screen, or not (remembered)
y
copy a permalink pinned to the commit
d u x
diff: split / unified / collapse or expand all
e E
diff: the whole file in view / every file, with the lines the hunks hide
n
merge request: next note in the diff
esc
close a dialog, leave a field

Branches 12

Tags 11

↑↓ moveenter openesc close