detached 0a1c4c5

feat(broker): compose offline broker process

commit
0a1c4c5f93642f9745aa34f8b4481e688b4b550f
author
brice · · 2026-09-16 22:24 UTC
parent
36c187d
signature
GPG VerifiedCD88F9E564E0102646FB6077A1A07C7C7C3E349D · arch-gpg Verified for the committer email
39 files · +8127 -87
Split
UnifiedSplit
M docs/architecture.md +187 -5
@@ -46,6 +46,13 @@ vev uses a hexagonal core with typed session messages at the client and daemon b
46 the short-lived SSH bootstrap (ephemeral certificate, 32-byte token,46 the short-lived SSH bootstrap (ephemeral certificate, 32-byte token,
47 nonce, ≤4 KiB readiness, ≤15 s expiry, atomic one-time consumption).47 nonce, ≤4 KiB readiness, ≤15 s expiry, atomic one-time consumption).
48 QUIC library types never leave the package.48 QUIC library types never leave the package.
49+- `internal/adapters/brokerconfig`: owns the strict isolated offline broker
50+ sandbox configuration (`config.json` marker), the textual offline-root
51+ validation that keeps the sandbox out of production runtime/state, and the
52+ immutable `ports.BrokerEndpointResolver` over the provisioned endpoint
53+ registrations, identities, policies, and absolute Unix mux routes. It is
54+ composed only by the hidden offline broker entry points (`_broker-serve`,
55+ `_broker-launcher`, and `_broker-status`).
49 - `internal/adapters/uidriver`: owns strict JSONL decoding, response serialization, bounded controller queues, private Unix sockets, peer credentials, and the stdio bridge. It consumes `ports.UIService` and never creates an attachment.56 - `internal/adapters/uidriver`: owns strict JSONL decoding, response serialization, bounded controller queues, private Unix sockets, peer credentials, and the stdio bridge. It consumes `ports.UIService` and never creates an attachment.
50 - `internal/adapters/webterm`: owns the authenticated loopback HTTP/WebSocket frontend, browser event encoding, VT-backed terminal and transactional HTML output. It implements `ports.Terminal`; application composition supplies the ordinary client runner.57 - `internal/adapters/webterm`: owns the authenticated loopback HTTP/WebSocket frontend, browser event encoding, VT-backed terminal and transactional HTML output. It implements `ports.Terminal`; application composition supplies the ordinary client runner.
51 - `internal/adapters/uiterm`: owns the immutable VT mirror/snapshot and deterministic headless terminal. The concrete terminal writer (`adapters/term`) taps successful writes and flushes into this sink only when observation is enabled.58 - `internal/adapters/uiterm`: owns the immutable VT mirror/snapshot and deterministic headless terminal. The concrete terminal writer (`adapters/term`) taps successful writes and flushes into this sink only when observation is enabled.
@@ -223,9 +230,23 @@ Every frame must carry the epoch and connection identity the listener assigned
223 at accept; a mismatched frame is fenced, and a stale stream open is answered230 at accept; a mismatched frame is fenced, and a stale stream open is answered
224 with a `StreamClosed` carrying the stale admission code. A malformed,231 with a `StreamClosed` carrying the stale admission code. A malformed,
225 oversize, or wrong-direction envelope settles exactly the connection that sent232 oversize, or wrong-direction envelope settles exactly the connection that sent
226-it. Operational bounds are explicit: concurrent clients per listener, per-stream233+it. Setup is fully bounded on both halves: the client dialer bounds the Unix
227-inbound chunks and bytes, pending mutating operations, and the brokerwire234+dial, the preamble, the Register send, and the wait for Registered with one
228-per-connection stream and dedup bounds. Disconnect cleanup is deterministic:235+context/deadline, closing the carriage to interrupt a peer that stalls after a
236+successful preamble and joining every setup worker, while a registered
237+connection detaches from that setup context; the server requires the client's
238+Register within the same accept-time handshake budget that bounded the preamble
239+and admission, so a silent same-user peer can neither pin an admitted core lease
240+nor hold a listener slot. Admission is the `ports.BrokerAuthority` seam,
241+implemented by `internal/usecase/broker.Authority`; the pre-Register guard
242+consults a registered marker set when Register handling begins, so a Register
243+arriving exactly at the deadline is admitted rather than settled as a timeout,
244+and a genuine registration timeout is classified as an orderly peer disconnect
245+so a listener draining its sessions never reports one as its own failure.
246+Operational bounds are explicit: concurrent clients
247+per listener, per-stream inbound chunks and bytes, pending mutating operations,
248+and the brokerwire per-connection stream and dedup bounds. Disconnect cleanup is
249+deterministic:
229 the session cancels its context, stops publishing, closes the outer carriage250 the session cancels its context, stops publishing, closes the outer carriage
230 (so a stream writer parked on a non-reading peer is interrupted), settles every251 (so a stream writer parked on a non-reading peer is interrupted), settles every
231 bridged stream (releasing its core connection and carriage), closes the252 bridged stream (releasing its core connection and carriage), closes the
@@ -235,6 +256,167 @@ reply is lost after the request was sent, and the client never replays them
235 blindly.256 blindly.
236 257
237 P3.3 introduces no production activation: nothing in `internal/app` or `main`258 P3.3 introduces no production activation: nothing in `internal/app` or `main`
238-constructs this adapter yet. P3.4 composes the broker use case, its stores, and259+constructs this adapter for ordinary operation. P3.4 composes the broker use
239-this listener behind a hidden entry point, and P7 performs the coordinated260+case, its stores, and this listener behind the hidden `_broker-serve` sandbox
261+entry point, with the listener consuming
262+`ports.BrokerAuthority` instead of defining its own seam. The composing caller
263+owns the `Registry.Run` lifecycle: it starts the single run to own and drain the
264+durable writer and cancels and joins that run before closing the store, because
265+`NewAuthority` neither starts nor settles the run. P7 performs the coordinated
240 cutover.266 cutover.
267+
268+## Broker offline sandbox (Plan 001 P3.4 slice C, not activated)
269+
270+The hidden `_broker-serve --offline-root ABS [--idle-grace DURATION]` entry
271+point is a foreground, isolated sandbox over one operator-supplied offline root.
272+It is excluded from public help and changes no ordinary command, path, or
273+factory. The root must be absolute, cleaned, symlink-free, and outside the
274+production runtime and state directories; the runtime, state, and log paths are
275+derived only beneath it and secured with `pkg/safedir.EnsurePrivate`, and the
276+durable store is opened with `brokerstore.OpenOffline` over the sandbox state
277+directory with no legacy inputs.
278+
279+`internal/adapters/brokerconfig` owns the strict sandbox configuration: a
280+bounded, owner-only `config.json` carrying the exact `vev.broker.offline/v1`
281+marker, with unknown fields, duplicate keys, trailing data, invalid UTF-8,
282+excessive nesting, and duplicate endpoints refused. Each immutable registration
283+provisions the exact daemon registration, the stable authenticated daemon
284+identity, the exact connection policy, and one absolute Unix mux route. The
285+resulting `ports.BrokerEndpointResolver` performs no I/O and invents no address:
286+a request must name a configured endpoint, carry its exact registration, and
287+request a compatible policy, so a request can never supply an address, identity,
288+secret, command, or policy of its own. Because the pool keys one physical
289+transport by the exact authenticated identity plus policy pair and deliberately
290+ignores the opaque route address, two registrations that share that pair must
291+agree on the one route the pool dials: an alias of an identical route is
292+accepted under distinct endpoints, while a second route for the same pair is
293+refused instead of being dialed for neither endpoint.
294+
295+The composition samples a cryptographically random non-zero broker epoch and
296+builds a store-backed, observation-disabled `broker.Registry` (whose single
297+`Run` the supervisor owns and joins), the immutable resolver, a
298+`daemonmux.EndpointConnector` over `ipc.DialMuxContext`, a bounded `Pool`, the
299+idle `Supervisor` (5m default grace), `broker.Authority`, and the
300+`brokeripc` listener. A temporary startup lease pins the supervisor open until
301+the listener and its accept drain are ready; the drain continuously takes
302+admitted sessions without closing them, because each returned service is the
303+listener's own connection-scoped session. A signal or parent-context
304+cancellation, or the idle grace, shuts down in the fixed order listener ->
305+pool -> registry -> store -> log -> owner lock. The foreground process holds a
306+reusable lifetime owner lock (`internal/adapters/lifecycle.TryAcquire`) on the
307+sandbox runtime directory for its whole run, so a second owner fails closed.
308+The optional `idleGrace` field in `config.json` provisions the sandbox's
309+effective idle grace as a Go duration string. It is the single deterministic
310+source for that grace: `_broker-serve` and `_broker-launcher` refuse an explicit
311+`--idle-grace` that conflicts with the provisioned value, because a status probe
312+cannot read a running broker's effective grace through the broker IPC protocol.
313+When no grace is provisioned, an explicit flag (or the supervisor default)
314+applies.
315+
316+## Broker offline connect-or-spawn and status (Plan 001 P3.4 slice D, not activated)
317+
318+The hidden `_broker-launcher --offline-root ABS [--idle-grace DURATION]` entry
319+point is the intermediate half of a double fork: it validates the sandbox
320+configuration (including the idle-grace conflict rule), starts one detached
321+`_broker-serve` in a new session, and exits, exactly like the daemon launcher.
322+The hidden `_broker-status --offline-root ABS [--ensure] [--timeout DURATION]`
323+entry point reports the offline broker's state. Both are excluded from public
324+help and change no ordinary command, path, or factory.
325+
326+Status without `--ensure` is dial-only: it performs one bounded probe of the
327+sandbox socket and prints exactly one bounded JSON document,
328+`{status, endpoint, epoch, revision, host_count}`, where `endpoint` is the
329+broker IPC socket path and the epoch, revision, and host count are zero when the
330+broker is offline. A missing socket and a stale socket left by a dead owner are
331+both recoverable absence; a foreign path, a live endpoint that fails the broker
332+handshake, and a stalled handshake are incompatible and never respawned over.
333+`--timeout` is the probe's single absolute bound in dial-only mode, so a caller
334+can tighten the probe or rely on the default (10s) and observe the same bound.
335+Dial-only status always exits zero, so the JSON is the sole result. `--ensure`
336+runs one absolute overall deadline (default 10s): it validates and secures the
337+sandbox root exactly as the launcher does before creating the spawn directory or
338+taking any lock. On success it prints the ready report and exits zero; on any
339+layout, configuration, or startup failure it still prints the offline report but
340+exits 3, so the caller always observes one offline JSON document and a distinct
341+exit code. The launcher is spawned under the ensure context
342+(`exec.CommandContext`), so the overall deadline or a signal bounds a launcher
343+that never exits; a launcher that already started the detached `_broker-serve`
344+is still safe to kill, because that serve is in its own session and survives.
345+
346+A foreground `_broker-serve` that hits an unexpected terminal accept failure
347+commits the same ordered shutdown as a cancellation, so a socket that can no
348+longer admit work is never left bound.
349+
350+Race safety rests on two descriptor-backed exclusive locks that are never
351+stolen by age and whose inodes are never unlinked. Both refuse any lock with
352+group or other access and trust an owner-only file whatever its exact owner bit
353+pattern, so a lock created under a restrictive umask (normalized to 0600 on
354+creation) or by a different tool never wedges startup; a group/other-readable
355+path stays refused. The broker lifetime lock
356+(`layout.Runtime/lifecycle.lock`) is held by the foreground `_broker-serve` for
357+its whole run; only the elected spawner probes it, and only as a liveness hint.
358+The spawn-election lock (`layout.Spawn/lifecycle.lock`, a separate private
359+directory beneath runtime) elects exactly one status caller to spawn. The
360+elected spawner re-dials after winning the lock, then holds the election until
361+the endpoint answers a valid broker registration and publishes its initial
362+snapshot, or until the deadline expires. A waiter never touches the lifetime
363+lock, so it can never briefly steal ownership from the broker it is waiting for,
364+and every wait is a bounded retry rather than a sleep loop. If the elected
365+spawner dies before readiness, the kernel releases its lock and a waiter takes
366+over; a duplicate `_broker-serve` fails its own lifetime acquisition
367+nonblocking. Every spawn uses the existing double-fork process pattern, and the
368+status probe and launcher inherit only the ambient environment minus the
369+performance-trace inputs.
370+
371+## Broker remote mux helpers (Plan 001 P3.4 slice E, not activated)
372+
373+The offline broker reaches a remote daemon's private daemonmux carriage over one
374+of three explicitly provisioned, bounded routes, and P3.4 slice E adds the
375+transport-selecting connector plus three hidden remote helpers. The routing
376+schema is strict and immutable: `internal/adapters/brokerconfig` parses a
377+registration's `route` as either the original Unix mux path string or an object
378+with an explicit kind, and refuses unknown kinds, unknown fields, trailing data,
379+and any kind/field mismatch. The bounded `ssh-stdio` and `ssh-quic` variants
380+carry an ssh target, an explicit remote argv, and optional trust inputs (a
381+known-hosts file and a connect timeout). A route can never carry an identity, a
382+policy, a command chosen at request time, or a secret. Identity and policy stay
383+authoritative on the registration, and the resolver publishes only an opaque
384+bounded route address: a digest of the route's own contents, so the pooling
385+layer never carries a target, a path, or an argv word.
386+
387+`_broker-serve` composes a transport-selecting `daemonmux.EndpointConnector`. Its
388+dial function receives only the opaque address, looks the route up in the
389+immutable configuration it was built with, and refuses an address the
390+configuration did not produce. A `unix` route dials the provisioned private Unix
391+mux carriage directly; an `ssh-stdio` route starts one explicit ssh child (built
392+with `sshstdio.BuildCommandForMux` and carrying only trust inputs that narrow
393+verification, never disabling host-key checking, forcing an auth bypass, or
394+requesting a PTY) through `sshstdio.DialMuxContext`; an `ssh-quic` route runs one
395+SSH-authenticated QUIC bootstrap, reads exactly one readiness line, and pins the
396+freshly minted ephemeral certificate through `quic.DialMuxContext`. One
397+caller-supplied setup context bounds bootstrap, pinned dial, and the daemonmux
398+physical handshake together; `brokerMuxSetupTimeout` is an independent backstop
399+for the broker-side bootstrap readiness wait and the child's kill/reap cleanup,
400+so an earlier caller deadline stays authoritative while a helper that never
401+prints readiness is still interrupted and reaped within the bound. The captured
402+bootstrap stderr sink is read only after that kill and reap, and the sink is
403+itself synchronized, so the os/exec copy goroutine never races the diagnostic. A
404+successful physical connection detaches from that context and stays pool-owned.
405+QUIC credentials are minted fresh per physical
406+connection and consumed exactly once, and captured bootstrap stderr is bounded
407+and sanitized, so a token, a nonce, or raw remote text never reaches an address,
408+a log, or an error.
409+
410+The three hidden helpers are excluded from public help. `_broker-mux-stdio
411+--offline-root ABS` validates the sandbox and bridges its own stdio to the single
412+provisioned private Unix daemonmux carriage. `_broker-mux-quic-bootstrap
413+--offline-root ABS` validates the sandbox, starts one detached
414+`_broker-mux-quic-proxy` in a new session, forwards its single readiness line,
415+and exits. `_broker-mux-quic-proxy --offline-root ABS` mints one ephemeral QUIC
416+server, admits exactly one authenticated carriage, and bridges it to the same
417+private Unix daemonmux carriage. A helper only ever bridges to the configured
418+private endpoint: it never starts a broker, an observer, or an ordinary daemon,
419+never dials the production daemon socket, and never fabricates a daemon
420+incarnation, which the daemon on the far side of the carriage alone owns.
421+Ordinary production composition still waits for P7; the helpers change no
422+ordinary command, path, or factory.
M docs/protocol.md +10 -1
@@ -117,7 +117,16 @@ composition until P3.2. The P3.3 private local endpoint
117 (`internal/adapters/brokeripc`) carries this conversation over one per-user117 (`internal/adapters/brokeripc`) carries this conversation over one per-user
118 AF_UNIX socket: `broker.sock` in the per-user runtime directory, with118 AF_UNIX socket: `broker.sock` in the per-user runtime directory, with
119 same-user peer-credential admission, bounded per-connection queues, and119 same-user peer-credential admission, bounded per-connection queues, and
120-production composition deferred to P3.4 and the P7 cutover.120+production composition deferred to P3.4 and the P7 cutover. P3.4 adds
121+the hidden, isolated offline sandbox entries -- `_broker-serve`,
122+`_broker-launcher`, and `_broker-status` -- over an operator-supplied private
123+root, plus the hidden remote mux helpers `_broker-mux-stdio`,
124+`_broker-mux-quic-bootstrap`, and `_broker-mux-quic-proxy` that bridge a
125+provisioned private daemonmux Unix carriage to stdio or to one freshly
126+authenticated QUIC stream; ordinary production composition still waits for P7.
127+Setup is fully bounded: the dialer's one context/deadline covers the Unix
128+dial, preamble, Register send, and wait for Registered, and the server requires Register within
129+the same accept-time handshake budget that bounded the preamble and admission.
121 130
122 ## Output, ACK, and flow control131 ## Output, ACK, and flow control
123 132
A internal/adapters/brokerconfig/config.go +607 -0
@@ -0,0 +1,607 @@
1+// Package brokerconfig owns the strict, isolated offline broker sandbox
2+// configuration (Plan 001 P3.4 slice C).
3+//
4+// It is deliberately not production composition: nothing in internal/app or
5+// main consults it for ordinary daemon operation. It resolves one caller-
6+// supplied offline root into sandbox-owned runtime, state, and log paths,
7+// validates that the root is a private, absolute, cleaned, symlink-free
8+// directory that does not overlap production runtime or state, and strictly
9+// parses the bounded config.json marker that provisions immutable endpoint
10+// registrations, each carrying a stable daemon identity, an exact connection
11+// policy, and one bounded route (a private Unix mux socket, an SSH stdio mux
12+// helper, or an SSH-authenticated QUIC bootstrap).
13+//
14+// The parsed configuration is immutable and is the only source of endpoint
15+// addresses, identities, and policies. A client request can name a configured
16+// endpoint and must carry that endpoint's exact registration and compatible
17+// policy; it can never supply an address, identity, secret, command, or policy
18+// of its own, and an unknown or stale request is refused without dialing. The
19+// route's pool address is an opaque digest, so no route detail is exposed to the
20+// pooling layer.
21+//
22+// Because the pool keys one physical transport by the exact authenticated
23+// identity plus policy pair and ignores that opaque address, two registrations
24+// that share the pair must agree on the one route the pool dials: distinct
25+// endpoints may alias an identical route, but a second route for the same pair
26+// is refused rather than silently stranded.
27+package brokerconfig
28+
29+import (
30+ "bytes"
31+ "encoding/hex"
32+ "encoding/json"
33+ "errors"
34+ "fmt"
35+ "io"
36+ "io/fs"
37+ "os"
38+ "path/filepath"
39+ "strings"
40+ "syscall"
41+ "time"
42+ "unicode/utf8"
43+
44+ "github.com/bnema/vev/internal/domain"
45+ "github.com/bnema/vev/internal/ports"
46+ "github.com/bnema/vev/internal/protocol"
47+)
48+
49+// Marker is the exact config.json marker that identifies a broker offline
50+// sandbox configuration. A file without it is never treated as broker config,
51+// so an accidental --offline-root pointing at unrelated JSON fails closed.
52+const Marker = "vev.broker.offline/v1"
53+
54+// ConfigFileName is the fixed config file name inside one offline root.
55+const ConfigFileName = "config.json"
56+
57+// MaxConfigBytes bounds the config file before any decoding.
58+const MaxConfigBytes = 1 << 20
59+
60+// MaxIdleGrace bounds a provisioned idle grace so an accidental enormous value
61+// cannot make an offline broker effectively immortal.
62+const MaxIdleGrace = 24 * time.Hour
63+
64+// Sandbox directory names derived from one offline root. SpawnDirName is the
65+// private directory under Runtime that holds the descriptor-backed
66+// spawn-election lock, deliberately separate from the broker lifetime lock in
67+// Runtime itself so electing a spawner can never collide with lifetime
68+// ownership.
69+const (
70+ RuntimeDirName = "runtime"
71+ StateDirName = "state"
72+ LogDirName = "log"
73+ SpawnDirName = "spawn"
74+)
75+
76+var (
77+ // errUnknownEndpoint reports a request for an endpoint the immutable
78+ // configuration never provisioned.
79+ errUnknownEndpoint = errors.New("brokerconfig: unknown broker endpoint")
80+ // errStaleRegistration reports a request whose registration no longer
81+ // matches the provisioned immutable registration.
82+ errStaleRegistration = errors.New("brokerconfig: stale broker registration")
83+ // errNoLocalRoute reports a local request: the offline sandbox provisions
84+ // remote Unix mux routes only, so it owns no local service binding.
85+ errNoLocalRoute = errors.New("brokerconfig: offline broker has no local route")
86+)
87+
88+// Layout names the sandbox-owned paths derived from one offline root. Every
89+// path is beneath Root; no production runtime or state path is ever derived. It
90+// also carries the reserved production directories it was validated against, so
91+// Load can fence every provisioned route against them too.
92+type Layout struct {
93+ Root string
94+ Runtime string
95+ State string
96+ Log string
97+ // Spawn is the private spawn-election directory beneath Runtime.
98+ Spawn string
99+
100+ // reserved holds the cleaned absolute production runtime and state paths
101+ // supplied to ResolveLayout. Load refuses a provisioned route that overlaps
102+ // them; it is unexported so only ResolveLayout can set it.
103+ reserved []string
104+}
105+
106+// ResolveLayout validates one offline root and derives its sandbox paths.
107+//
108+// The root must be absolute and clean, must not be the filesystem root, must
109+// not contain a symlinked component, and must not equal, contain, or sit inside
110+// any reserved production directory. Reserved directories are compared cleaned
111+// and absolute; empty entries are ignored. The result is purely textual: no
112+// directory is created here, and the caller secures the paths with
113+// pkg/safedir.EnsurePrivate.
114+func ResolveLayout(rawRoot string, reserved []string) (Layout, error) {
115+ if rawRoot == "" {
116+ return Layout{}, errors.New("brokerconfig: offline root is empty")
117+ }
118+ if !filepath.IsAbs(rawRoot) {
119+ return Layout{}, fmt.Errorf("brokerconfig: offline root %q is not absolute", rawRoot)
120+ }
121+ if cleaned := filepath.Clean(rawRoot); cleaned != rawRoot {
122+ return Layout{}, fmt.Errorf("brokerconfig: offline root %q is not clean", rawRoot)
123+ }
124+ if rawRoot == string(filepath.Separator) {
125+ return Layout{}, errors.New("brokerconfig: offline root must not be the filesystem root")
126+ }
127+ if err := rejectSymlinkedComponents(rawRoot); err != nil {
128+ return Layout{}, err
129+ }
130+ for _, other := range reserved {
131+ if other == "" {
132+ continue
133+ }
134+ other = filepath.Clean(other)
135+ if !filepath.IsAbs(other) {
136+ return Layout{}, fmt.Errorf("brokerconfig: reserved directory %q is not absolute", other)
137+ }
138+ if overlaps(rawRoot, other) {
139+ return Layout{}, fmt.Errorf("brokerconfig: offline root %q overlaps production path %q", rawRoot, other)
140+ }
141+ }
142+ runtime := filepath.Join(rawRoot, RuntimeDirName)
143+ return Layout{
144+ Root: rawRoot,
145+ Runtime: runtime,
146+ State: filepath.Join(rawRoot, StateDirName),
147+ Log: filepath.Join(rawRoot, LogDirName),
148+ Spawn: filepath.Join(runtime, SpawnDirName),
149+ reserved: append([]string(nil), reserved...),
150+ }, nil
151+}
152+
153+// VerifyCreated re-walks every sandbox-owned path and refuses any existing
154+// component that is a symlink.
155+//
156+// ResolveLayout validates the root textually and stops at the first component
157+// that does not exist yet, so the composing caller calls this once
158+// pkg/safedir.EnsurePrivate has created the runtime, state, and log
159+// directories. It closes the gap between that validation and the first use of
160+// the created paths: a component swapped for a symlink in the window fails
161+// closed instead of being followed.
162+func (l Layout) VerifyCreated() error {
163+ for _, path := range []string{l.Root, l.Runtime, l.State, l.Log, l.Spawn} {
164+ if err := rejectSymlinkedComponents(path); err != nil {
165+ return err
166+ }
167+ }
168+ return nil
169+}
170+
171+// rejectSymlinkedComponents refuses a path any of whose existing components is
172+// a symlink. A component that does not exist yet ends the walk: the caller
173+// creates the remaining private directories itself.
174+func rejectSymlinkedComponents(abs string) error {
175+ volume := filepath.VolumeName(abs)
176+ rest := strings.TrimPrefix(abs, volume)
177+ rest = strings.TrimPrefix(rest, string(filepath.Separator))
178+ current := volume + string(filepath.Separator)
179+ if rest == "" {
180+ return nil
181+ }
182+ for _, part := range strings.Split(rest, string(filepath.Separator)) {
183+ if part == "" {
184+ continue
185+ }
186+ current = filepath.Join(current, part)
187+ info, err := os.Lstat(current)
188+ if err != nil {
189+ if errors.Is(err, fs.ErrNotExist) {
190+ return nil
191+ }
192+ return fmt.Errorf("brokerconfig: inspect %q: %w", current, err)
193+ }
194+ if info.Mode()&os.ModeSymlink != 0 {
195+ return fmt.Errorf("brokerconfig: %q is a symlink", current)
196+ }
197+ }
198+ return nil
199+}
200+
201+// overlaps reports whether two absolute cleaned paths are equal or one contains
202+// the other.
203+func overlaps(a, b string) bool {
204+ if a == b {
205+ return true
206+ }
207+ sep := string(filepath.Separator)
208+ return strings.HasPrefix(a, b+sep) || strings.HasPrefix(b, a+sep)
209+}
210+
211+// Registration is one immutable, provisioned endpoint: the exact daemon
212+// registration a request must carry, the stable authenticated daemon identity,
213+// the exact connection policy, and the bounded route the connector dials.
214+type Registration struct {
215+ Registration domain.RemoteRegistration
216+ Identity ports.BrokerDaemonIdentity
217+ Policy ports.BrokerPolicy
218+ Route Route
219+}
220+
221+// pooledIdentity is the exact (authenticated identity, policy) pair the broker
222+// pool keys one physical transport by. The pool deliberately ignores the opaque
223+// route address, so two registrations that share a pair share one pooled
224+// transport and must therefore agree on the single route that transport dials.
225+type pooledIdentity struct {
226+ identity ports.BrokerDaemonIdentity
227+ policy ports.BrokerPolicy
228+}
229+
230+// Config is the validated, immutable offline broker configuration.
231+type Config struct {
232+ byEndpoint map[string]Registration
233+ byAddress map[string]Route
234+ unixRoutes []Route
235+ order []string
236+ // idleGrace is the provisioned effective idle grace; hasIdleGrace records
237+ // whether the file provisioned one, so a caller can distinguish "no
238+ // provisioning, use the built-in default" from "provisioned".
239+ idleGrace time.Duration
240+ hasIdleGrace bool
241+}
242+
243+// Load reads and strictly validates the config marker inside layout.Root.
244+//
245+// The file must be a regular owner-only file that is not a symlink and is
246+// bounded by MaxConfigBytes; the JSON must carry the exact marker, reject
247+// unknown fields, duplicate keys, trailing data, invalid UTF-8, and excessive
248+// nesting. Every registration is validated independently and endpoints must be
249+// unique. Two registrations may alias one identical route (the same opaque
250+// address) under different endpoints, but two that share the pool's exact
251+// (authenticated identity, policy) pair must agree on that one route: the pool
252+// keys a physical transport by the pair and ignores the address, so a second
253+// route for the same pair is refused rather than silently dialed or ignored. A
254+// missing or malformed config is an error, never an empty default.
255+func Load(layout Layout) (*Config, error) {
256+ raw, err := readConfig(filepath.Join(layout.Root, ConfigFileName))
257+ if err != nil {
258+ return nil, err
259+ }
260+ var document configDocument
261+ if err := decodeStrict(raw, &document); err != nil {
262+ return nil, fmt.Errorf("brokerconfig: %s: %w", ConfigFileName, err)
263+ }
264+ if document.Marker != Marker {
265+ return nil, fmt.Errorf("brokerconfig: %s: missing or invalid marker", ConfigFileName)
266+ }
267+ config := &Config{
268+ byEndpoint: make(map[string]Registration, len(document.Registrations)),
269+ byAddress: make(map[string]Route, len(document.Registrations)),
270+ order: make([]string, 0, len(document.Registrations)),
271+ }
272+ if document.IdleGrace != "" {
273+ grace, err := time.ParseDuration(document.IdleGrace)
274+ if err != nil {
275+ return nil, fmt.Errorf("brokerconfig: %s: idleGrace %q is not a duration", ConfigFileName, document.IdleGrace)
276+ }
277+ if grace <= 0 || grace > MaxIdleGrace {
278+ return nil, fmt.Errorf("brokerconfig: %s: idleGrace must be positive and at most %s", ConfigFileName, MaxIdleGrace)
279+ }
280+ config.idleGrace, config.hasIdleGrace = grace, true
281+ }
282+ pooledRoutes := make(map[pooledIdentity]string, len(document.Registrations))
283+ for i, entry := range document.Registrations {
284+ registration, err := entry.validate()
285+ if err != nil {
286+ return nil, fmt.Errorf("brokerconfig: registration %d: %w", i, err)
287+ }
288+ if registration.Route.IsLocal() {
289+ if err := rejectRouteOverlap(registration.Route.Path(), layout.reserved); err != nil {
290+ return nil, fmt.Errorf("brokerconfig: registration %d: %w", i, err)
291+ }
292+ }
293+ if _, duplicate := config.byEndpoint[registration.Registration.Endpoint]; duplicate {
294+ return nil, fmt.Errorf("brokerconfig: registration %d: duplicate endpoint %q", i, registration.Registration.Endpoint)
295+ }
296+ // The pool keys one physical transport by (identity, policy) and ignores
297+ // the route address, so a second address for the same pair would be
298+ // dialed for neither or both depending on scheduling. Refuse it and allow
299+ // an alias only when the routes are identical.
300+ key := pooledIdentity{identity: registration.Identity, policy: registration.Policy}
301+ address := registration.Route.Address()
302+ if existing, present := pooledRoutes[key]; present && existing != address {
303+ return nil, fmt.Errorf("brokerconfig: registration %d: identity %q and policy already provisioned on a different route", i, registration.Identity)
304+ }
305+ pooledRoutes[key] = address
306+ config.byEndpoint[registration.Registration.Endpoint] = registration
307+ config.byAddress[address] = registration.Route
308+ if registration.Route.IsLocal() {
309+ config.unixRoutes = appendUnixRouteOnce(config.unixRoutes, registration.Route)
310+ }
311+ config.order = append(config.order, registration.Registration.Endpoint)
312+ }
313+ return config, nil
314+}
315+
316+// appendUnixRouteOnce adds a local Unix route unless an identical route (the
317+// same opaque address, hence the same bounded contents) is already present, so
318+// endpoint aliases of one route never make LocalMuxRoute report more than one
319+// provisioned route.
320+func appendUnixRouteOnce(routes []Route, route Route) []Route {
321+ for _, existing := range routes {
322+ if existing.Address() == route.Address() {
323+ return routes
324+ }
325+ }
326+ return append(routes, route)
327+}
328+
329+// Endpoints returns the configured endpoints in file order. The result is a
330+// copy; the configuration is immutable.
331+func (c *Config) Endpoints() []string {
332+ if c == nil {
333+ return nil
334+ }
335+ return append([]string(nil), c.order...)
336+}
337+
338+// IdleGrace returns the provisioned effective idle grace and whether the
339+// configuration provisioned one. A nil configuration provisions nothing.
340+func (c *Config) IdleGrace() (time.Duration, bool) {
341+ if c == nil {
342+ return 0, false
343+ }
344+ return c.idleGrace, c.hasIdleGrace
345+}
346+
347+// RouteByAddress resolves one opaque pool address back to the route it was
348+// derived from. The address is produced only by this configuration's own
349+// routes, so an unknown address is refused rather than dialed.
350+func (c *Config) RouteByAddress(address string) (Route, bool) {
351+ if c == nil {
352+ return Route{}, false
353+ }
354+ route, ok := c.byAddress[address]
355+ return route, ok
356+}
357+
358+// LocalMuxRoute returns the single provisioned Unix route a remote-side mux
359+// helper bridges to. A configuration that provisions no Unix route, or more
360+// than one, is refused: a helper must never guess which daemon-mux endpoint it
361+// serves.
362+func (c *Config) LocalMuxRoute() (Route, error) {
363+ if c == nil {
364+ return Route{}, errors.New("brokerconfig: no configuration")
365+ }
366+ switch len(c.unixRoutes) {
367+ case 0:
368+ return Route{}, errors.New("brokerconfig: no Unix daemon-mux route is provisioned")
369+ case 1:
370+ return c.unixRoutes[0], nil
371+ default:
372+ return Route{}, errors.New("brokerconfig: more than one Unix daemon-mux route is provisioned")
373+ }
374+}
375+
376+// Resolver returns the immutable endpoint resolver over this configuration.
377+func (c *Config) Resolver() *Resolver {
378+ if c == nil {
379+ return &Resolver{byEndpoint: map[string]Registration{}}
380+ }
381+ return &Resolver{byEndpoint: c.byEndpoint}
382+}
383+
384+// configDocument is the strict on-disk shape. Unknown fields are refused by the
385+// decoder, so the file cannot smuggle additional behavior.
386+type configDocument struct {
387+ Marker string `json:"marker"`
388+ // IdleGrace optionally provisions the sandbox's effective idle grace as a Go
389+ // duration string (for example "90s"). It is the single deterministic source
390+ // for the idle grace, because a status probe cannot read a running broker's
391+ // effective grace through the broker IPC protocol.
392+ IdleGrace string `json:"idleGrace,omitempty"`
393+ Registrations []registrationRaw `json:"registrations"`
394+}
395+
396+// registrationRaw is the raw JSON registration before semantic validation.
397+type registrationRaw struct {
398+ Endpoint string `json:"endpoint"`
399+ Incarnation string `json:"incarnation"`
400+ Generation uint64 `json:"generation"`
401+ Identity string `json:"identity"`
402+ Route json.RawMessage `json:"route"`
403+ Policy policyRaw `json:"policy"`
404+}
405+
406+// policyRaw is the raw JSON policy before semantic validation.
407+type policyRaw struct {
408+ ProtocolVersion uint16 `json:"protocolVersion"`
409+ CatalogSchemaVersion uint16 `json:"catalogSchemaVersion"`
410+ EnvironmentPolicy string `json:"environmentPolicy"`
411+ Transport string `json:"transport"`
412+ Trust string `json:"trust"`
413+ Launch string `json:"launch"`
414+ Isolation string `json:"isolation"`
415+}
416+
417+func (r registrationRaw) validate() (Registration, error) {
418+ var registration domain.RemoteRegistration
419+ registration.Endpoint = r.Endpoint
420+ if err := domain.ValidateRemoteHostTarget(r.Endpoint); err != nil {
421+ return Registration{}, err
422+ }
423+ if len(r.Incarnation) != 32 {
424+ return Registration{}, errors.New("incarnation must be 32 hexadecimal characters")
425+ }
426+ decoded, err := hex.DecodeString(r.Incarnation)
427+ if err != nil {
428+ return Registration{}, fmt.Errorf("incarnation: %w", err)
429+ }
430+ copy(registration.Incarnation[:], decoded)
431+ registration.Generation = domain.RemoteGeneration(r.Generation)
432+ if err := registration.Validate(); err != nil {
433+ return Registration{}, err
434+ }
435+ identity := ports.BrokerDaemonIdentity(r.Identity)
436+ if err := identity.Validate(); err != nil {
437+ return Registration{}, err
438+ }
439+ policy, err := r.Policy.validate()
440+ if err != nil {
441+ return Registration{}, err
442+ }
443+ route, err := parseRoute(r.Route)
444+ if err != nil {
445+ return Registration{}, err
446+ }
447+ return Registration{Registration: registration, Identity: identity, Policy: policy, Route: route}, nil
448+}
449+
450+func (p policyRaw) validate() (ports.BrokerPolicy, error) {
451+ var environment protocol.EnvironmentPolicy
452+ switch p.EnvironmentPolicy {
453+ case "client-owned":
454+ environment = protocol.EnvironmentPolicyClientOwned
455+ case "daemon-owned":
456+ environment = protocol.EnvironmentPolicyDaemonOwned
457+ default:
458+ return ports.BrokerPolicy{}, fmt.Errorf("policy environmentPolicy %q is invalid", p.EnvironmentPolicy)
459+ }
460+ policy := ports.BrokerPolicy{
461+ ProtocolVersion: p.ProtocolVersion,
462+ CatalogSchemaVersion: p.CatalogSchemaVersion,
463+ EnvironmentPolicy: environment,
464+ Transport: p.Transport,
465+ Trust: p.Trust,
466+ Launch: p.Launch,
467+ Isolation: p.Isolation,
468+ }
469+ if err := policy.Validate(); err != nil {
470+ return ports.BrokerPolicy{}, err
471+ }
472+ return policy, nil
473+}
474+
475+// rejectRouteOverlap refuses a provisioned Unix mux route that equals, contains,
476+// or sits inside a reserved production runtime or state directory. A route is an
477+// address the broker connector dials, so a route overlapping production would
478+// let the isolated offline sandbox reach a production endpoint even though the
479+// sandbox root itself is outside production. Reserved entries are cleaned
480+// absolute paths; empty entries are ignored.
481+func rejectRouteOverlap(route string, reserved []string) error {
482+ for _, other := range reserved {
483+ if other == "" {
484+ continue
485+ }
486+ other = filepath.Clean(other)
487+ if overlaps(route, other) {
488+ return fmt.Errorf("route %q overlaps production path %q", route, other)
489+ }
490+ }
491+ return nil
492+}
493+
494+// validateRoute refuses anything but an absolute, cleaned Unix path without
495+// symlinked components, so a provisioned route can never be reshaped by the
496+// request path or escape through a link.
497+func validateRoute(route string) error {
498+ if route == "" {
499+ return errors.New("route is empty")
500+ }
501+ if !filepath.IsAbs(route) {
502+ return fmt.Errorf("route %q is not absolute", route)
503+ }
504+ if filepath.Clean(route) != route {
505+ return fmt.Errorf("route %q is not clean", route)
506+ }
507+ return rejectSymlinkedComponents(route)
508+}
509+
510+// readConfig reads one bounded regular owner-only config file without following
511+// a symlink.
512+func readConfig(path string) ([]byte, error) {
513+ fd, err := syscall.Open(path, syscall.O_RDONLY|syscall.O_CLOEXEC|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0)
514+ if err != nil {
515+ if errors.Is(err, syscall.ELOOP) {
516+ return nil, fmt.Errorf("brokerconfig: %s is a symlink", path)
517+ }
518+ return nil, fmt.Errorf("brokerconfig: open %s: %w", path, err)
519+ }
520+ file := os.NewFile(uintptr(fd), path)
521+ defer file.Close()
522+ info, err := file.Stat()
523+ if err != nil {
524+ return nil, fmt.Errorf("brokerconfig: stat %s: %w", path, err)
525+ }
526+ if !info.Mode().IsRegular() {
527+ return nil, fmt.Errorf("brokerconfig: %s is not a regular file", path)
528+ }
529+ if info.Mode().Perm()&0o077 != 0 {
530+ return nil, fmt.Errorf("brokerconfig: %s has permissions %04o, want owner-only", path, info.Mode().Perm())
531+ }
532+ if info.Size() > MaxConfigBytes {
533+ return nil, fmt.Errorf("brokerconfig: %s exceeds %d bytes", path, MaxConfigBytes)
534+ }
535+ raw, err := io.ReadAll(io.LimitReader(file, MaxConfigBytes+1))
536+ if err != nil {
537+ return nil, fmt.Errorf("brokerconfig: read %s: %w", path, err)
538+ }
539+ if len(raw) > MaxConfigBytes {
540+ return nil, fmt.Errorf("brokerconfig: %s exceeds %d bytes", path, MaxConfigBytes)
541+ }
542+ return raw, nil
543+}
544+
545+// decodeStrict rejects duplicate keys, unknown fields, trailing JSON, invalid
546+// UTF-8, and excessive nesting, then decodes once. Byte bounds are applied
547+// before decoding.
548+func decodeStrict(raw []byte, dst any) error {
549+ if len(raw) > MaxConfigBytes || !utf8.Valid(raw) {
550+ return errors.New("invalid JSON size or encoding")
551+ }
552+ scan := json.NewDecoder(bytes.NewReader(raw))
553+ if err := scanJSON(scan, 0); err != nil {
554+ return err
555+ }
556+ if _, err := scan.Token(); err != io.EOF {
557+ return errors.New("trailing JSON")
558+ }
559+ decoder := json.NewDecoder(bytes.NewReader(raw))
560+ decoder.DisallowUnknownFields()
561+ return decoder.Decode(dst)
562+}
563+
564+// scanJSON walks one JSON value, rejecting duplicate keys and excessive depth.
565+func scanJSON(decoder *json.Decoder, depth int) error {
566+ if depth > 64 {
567+ return errors.New("JSON nesting limit")
568+ }
569+ token, err := decoder.Token()
570+ if err != nil {
571+ return err
572+ }
573+ delimiter, ok := token.(json.Delim)
574+ if !ok {
575+ return nil
576+ }
577+ switch delimiter {
578+ case '{':
579+ seen := make(map[string]bool)
580+ for decoder.More() {
581+ key, err := decoder.Token()
582+ if err != nil {
583+ return err
584+ }
585+ name, ok := key.(string)
586+ if !ok || seen[name] {
587+ return errors.New("duplicate JSON key")
588+ }
589+ seen[name] = true
590+ if err := scanJSON(decoder, depth+1); err != nil {
591+ return err
592+ }
593+ }
594+ _, err = decoder.Token()
595+ return err
596+ case '[':
597+ for decoder.More() {
598+ if err := scanJSON(decoder, depth+1); err != nil {
599+ return err
600+ }
601+ }
602+ _, err = decoder.Token()
603+ return err
604+ default:
605+ return errors.New("unexpected JSON delimiter")
606+ }
607+}
A internal/adapters/brokerconfig/config_test.go +653 -0
@@ -0,0 +1,653 @@
1+package brokerconfig
2+
3+import (
4+ "context"
5+ "encoding/json"
6+ "os"
7+ "path/filepath"
8+ "strings"
9+ "testing"
10+ "time"
11+
12+ "github.com/stretchr/testify/require"
13+
14+ "github.com/bnema/vev/internal/domain"
15+ "github.com/bnema/vev/internal/ports"
16+ "github.com/bnema/vev/internal/protocol"
17+)
18+
19+const (
20+ testIncarnation = "0102030405060708090a0b0c0d0e0f10"
21+ testEndpoint = "offline@daemon:2222"
22+)
23+
24+// testPolicy is one fully valid, exact policy for the offline sandbox.
25+func testPolicy() ports.BrokerPolicy {
26+ return ports.BrokerPolicy{
27+ ProtocolVersion: protocol.Version,
28+ CatalogSchemaVersion: 3,
29+ EnvironmentPolicy: protocol.EnvironmentPolicyClientOwned,
30+ Transport: "unix-mux",
31+ Trust: "offline-trust",
32+ Launch: "offline-launch",
33+ Isolation: "offline-isolation",
34+ }
35+}
36+
37+// registrationDocument renders one registration with the supplied overrides.
38+func registrationDocument(overrides map[string]any) map[string]any {
39+ entry := map[string]any{
40+ "endpoint": testEndpoint,
41+ "incarnation": testIncarnation,
42+ "generation": 1,
43+ "identity": "offline-daemon",
44+ "route": "/tmp/vev-brokerconfig-test/mux.sock",
45+ "policy": map[string]any{
46+ "protocolVersion": testPolicy().ProtocolVersion,
47+ "catalogSchemaVersion": testPolicy().CatalogSchemaVersion,
48+ "environmentPolicy": "client-owned",
49+ "transport": testPolicy().Transport,
50+ "trust": testPolicy().Trust,
51+ "launch": testPolicy().Launch,
52+ "isolation": testPolicy().Isolation,
53+ },
54+ }
55+ for key, value := range overrides {
56+ entry[key] = value
57+ }
58+ return map[string]any{
59+ "marker": Marker,
60+ "registrations": []any{entry},
61+ }
62+}
63+
64+// writeConfig marshals one document into root/config.json with owner-only
65+// permissions.
66+func writeConfig(t *testing.T, root string, document any, perm os.FileMode) {
67+ t.Helper()
68+ raw, err := json.Marshal(document)
69+ require.NoError(t, err)
70+ require.NoError(t, os.WriteFile(filepath.Join(root, ConfigFileName), raw, perm))
71+}
72+
73+func TestResolveLayout(t *testing.T) {
74+ base := t.TempDir()
75+ linkParent := filepath.Join(t.TempDir(), "link")
76+ require.NoError(t, os.Symlink(base, linkParent))
77+
78+ tests := []struct {
79+ name string
80+ root string
81+ reserved []string
82+ wantErr string
83+ }{
84+ {name: "valid absolute clean root", root: filepath.Join(base, "sandbox")},
85+ {name: "empty root", root: "", wantErr: "empty"},
86+ {name: "relative root", root: "sandbox", wantErr: "not absolute"},
87+ {name: "unclean root", root: base + "/sandbox/../sandbox", wantErr: "not clean"},
88+ {name: "filesystem root", root: string(filepath.Separator), wantErr: "filesystem root"},
89+ {name: "symlinked component", root: filepath.Join(linkParent, "sandbox"), wantErr: "symlink"},
90+ {name: "reserved equals root", root: filepath.Join(base, "sandbox"), reserved: []string{filepath.Join(base, "sandbox")}, wantErr: "overlaps"},
91+ {name: "reserved is parent", root: filepath.Join(base, "sandbox"), reserved: []string{base}, wantErr: "overlaps"},
92+ {name: "reserved is child", root: filepath.Join(base, "sandbox"), reserved: []string{filepath.Join(base, "sandbox", "state")}, wantErr: "overlaps"},
93+ {name: "reserved relative", root: filepath.Join(base, "sandbox"), reserved: []string{"relative"}, wantErr: "not absolute"},
94+ {name: "unrelated reserved is allowed", root: filepath.Join(base, "sandbox"), reserved: []string{filepath.Join(base, "other")}},
95+ }
96+ for _, tt := range tests {
97+ t.Run(tt.name, func(t *testing.T) {
98+ layout, err := ResolveLayout(tt.root, tt.reserved)
99+ if tt.wantErr != "" {
100+ require.Error(t, err)
101+ require.Contains(t, err.Error(), tt.wantErr)
102+ return
103+ }
104+ require.NoError(t, err)
105+ require.Equal(t, layout.Root, tt.root)
106+ require.Equal(t, filepath.Join(tt.root, RuntimeDirName), layout.Runtime)
107+ require.Equal(t, filepath.Join(tt.root, StateDirName), layout.State)
108+ require.Equal(t, filepath.Join(tt.root, LogDirName), layout.Log)
109+ })
110+ }
111+}
112+
113+func TestLoadValidConfig(t *testing.T) {
114+ root := t.TempDir()
115+ writeConfig(t, root, registrationDocument(nil), 0o600)
116+ layout, err := ResolveLayout(root, nil)
117+ require.NoError(t, err)
118+
119+ config, err := Load(layout)
120+ require.NoError(t, err)
121+ require.Equal(t, []string{testEndpoint}, config.Endpoints())
122+
123+ resolved, err := config.Resolver().Resolve(context.Background(), ports.BrokerOpenStreamRequest{
124+ Purpose: ports.BrokerStreamControl,
125+ Endpoint: testEndpoint,
126+ Registration: testRegistration(),
127+ Policy: testPolicy(),
128+ })
129+ require.NoError(t, err)
130+ require.Equal(t, ports.BrokerDaemonIdentity("offline-daemon"), resolved.Identity)
131+ require.Equal(t, testPolicy(), resolved.Policy)
132+
133+ local, err := config.LocalMuxRoute()
134+ require.NoError(t, err)
135+ require.Equal(t, RouteUnix, local.Kind())
136+ require.Equal(t, "/tmp/vev-brokerconfig-test/mux.sock", local.Path())
137+ require.Equal(t, local.Address(), resolved.Address)
138+ require.LessOrEqual(t, len(resolved.Address), MaxRouteAddressBytes)
139+ route, ok := config.RouteByAddress(resolved.Address)
140+ require.True(t, ok)
141+ require.Equal(t, local.Path(), route.Path())
142+ _, ok = config.RouteByAddress("offline-route-unknown")
143+ require.False(t, ok)
144+}
145+
146+func TestLoadAcceptsEmptyRegistrations(t *testing.T) {
147+ root := t.TempDir()
148+ writeConfig(t, root, map[string]any{"marker": Marker, "registrations": []any{}}, 0o600)
149+ layout, err := ResolveLayout(root, nil)
150+ require.NoError(t, err)
151+ config, err := Load(layout)
152+ require.NoError(t, err)
153+ require.Empty(t, config.Endpoints())
154+}
155+
156+func TestLoadRejects(t *testing.T) {
157+ tests := []struct {
158+ name string
159+ build func(t *testing.T, root string)
160+ wantErr string
161+ }{
162+ {
163+ name: "missing file",
164+ build: func(*testing.T, string) {},
165+ wantErr: "open",
166+ },
167+ {
168+ name: "wrong marker",
169+ build: func(t *testing.T, root string) {
170+ writeConfig(t, root, map[string]any{"marker": "other", "registrations": []any{}}, 0o600)
171+ },
172+ wantErr: "marker",
173+ },
174+ {
175+ name: "unknown field",
176+ build: func(t *testing.T, root string) {
177+ document := registrationDocument(nil)
178+ document["extra"] = true
179+ writeConfig(t, root, document, 0o600)
180+ },
181+ wantErr: "unknown field",
182+ },
183+ {
184+ name: "duplicate key",
185+ build: func(t *testing.T, root string) {
186+ writeRawConfig(t, root, `{"marker":"`+Marker+`","marker":"`+Marker+`"}`)
187+ },
188+ wantErr: "duplicate JSON key",
189+ },
190+ {
191+ name: "trailing json",
192+ build: func(t *testing.T, root string) {
193+ writeRawConfig(t, root, `{"marker":"`+Marker+`","registrations":[]} {}`)
194+ },
195+ wantErr: "trailing JSON",
196+ },
197+ {
198+ name: "invalid utf8",
199+ build: func(t *testing.T, root string) { writeRawConfig(t, root, "{\"marker\":\"\xff\"}") },
200+ wantErr: "encoding",
201+ },
202+ {
203+ name: "group readable",
204+ build: func(t *testing.T, root string) { writeConfig(t, root, registrationDocument(nil), 0o640) },
205+ wantErr: "owner-only",
206+ },
207+ {
208+ name: "symlinked config",
209+ build: func(t *testing.T, root string) {
210+ target := filepath.Join(t.TempDir(), "real.json")
211+ raw, err := json.Marshal(registrationDocument(nil))
212+ require.NoError(t, err)
213+ require.NoError(t, os.WriteFile(target, raw, 0o600))
214+ require.NoError(t, os.Symlink(target, filepath.Join(root, ConfigFileName)))
215+ },
216+ wantErr: "symlink",
217+ },
218+ {
219+ name: "nonregular config",
220+ build: func(t *testing.T, root string) {
221+ require.NoError(t, os.Mkdir(filepath.Join(root, ConfigFileName), 0o700))
222+ },
223+ wantErr: "not a regular file",
224+ },
225+ {
226+ name: "oversized config",
227+ build: func(t *testing.T, root string) {
228+ require.NoError(t, os.WriteFile(filepath.Join(root, ConfigFileName), make([]byte, MaxConfigBytes+1), 0o600))
229+ },
230+ wantErr: "exceeds",
231+ },
232+ {
233+ name: "symlinked route",
234+ build: func(t *testing.T, root string) {
235+ link := filepath.Join(t.TempDir(), "route-link")
236+ require.NoError(t, os.Symlink(t.TempDir(), link))
237+ writeConfig(t, root, registrationDocument(map[string]any{"route": filepath.Join(link, "mux.sock")}), 0o600)
238+ },
239+ wantErr: "symlink",
240+ },
241+ {
242+ name: "bad endpoint",
243+ build: func(t *testing.T, root string) {
244+ writeConfig(t, root, registrationDocument(map[string]any{"endpoint": "bad host"}), 0o600)
245+ },
246+ wantErr: "whitespace",
247+ },
248+ {
249+ name: "incarnation length",
250+ build: func(t *testing.T, root string) {
251+ writeConfig(t, root, registrationDocument(map[string]any{"incarnation": "abcd"}), 0o600)
252+ },
253+ wantErr: "32 hexadecimal",
254+ },
255+ {
256+ name: "incarnation zero",
257+ build: func(t *testing.T, root string) {
258+ writeConfig(t, root, registrationDocument(map[string]any{"incarnation": strings.Repeat("0", 32)}), 0o600)
259+ },
260+ wantErr: "no incarnation",
261+ },
262+ {
263+ name: "generation zero",
264+ build: func(t *testing.T, root string) {
265+ writeConfig(t, root, registrationDocument(map[string]any{"generation": 0}), 0o600)
266+ },
267+ wantErr: "no generation",
268+ },
269+ {
270+ name: "empty identity",
271+ build: func(t *testing.T, root string) {
272+ writeConfig(t, root, registrationDocument(map[string]any{"identity": ""}), 0o600)
273+ },
274+ wantErr: "identity",
275+ },
276+ {
277+ name: "bad environment policy",
278+ build: func(t *testing.T, root string) {
279+ writeConfig(t, root, documentWithPolicy(t, "environmentPolicy", "shared"), 0o600)
280+ },
281+ wantErr: "environmentPolicy",
282+ },
283+ {
284+ name: "zero protocol version",
285+ build: func(t *testing.T, root string) {
286+ writeConfig(t, root, documentWithPolicy(t, "protocolVersion", 0), 0o600)
287+ },
288+ wantErr: "no protocol version",
289+ },
290+ {
291+ name: "relative route",
292+ build: func(t *testing.T, root string) {
293+ writeConfig(t, root, registrationDocument(map[string]any{"route": "mux.sock"}), 0o600)
294+ },
295+ wantErr: "not absolute",
296+ },
297+ {
298+ name: "duplicate endpoint",
299+ build: func(t *testing.T, root string) {
300+ document := registrationDocument(nil)
301+ registrations := document["registrations"].([]any)
302+ document["registrations"] = append(registrations, registrations[0])
303+ writeConfig(t, root, document, 0o600)
304+ },
305+ wantErr: "duplicate endpoint",
306+ },
307+ }
308+ for _, tt := range tests {
309+ t.Run(tt.name, func(t *testing.T) {
310+ root := t.TempDir()
311+ tt.build(t, root)
312+ layout, err := ResolveLayout(root, nil)
313+ require.NoError(t, err)
314+ _, err = Load(layout)
315+ require.Error(t, err)
316+ require.Contains(t, err.Error(), tt.wantErr)
317+ })
318+ }
319+}
320+
321+func TestLoadRejectsRouteOverlappingProduction(t *testing.T) {
322+ prodRuntime := t.TempDir()
323+ prodState := t.TempDir()
324+
325+ tests := []struct {
326+ name string
327+ route string
328+ wantErr bool
329+ }{
330+ {name: "route under production runtime", route: filepath.Join(prodRuntime, "broker.sock"), wantErr: true},
331+ {name: "route equals production state", route: prodState, wantErr: true},
332+ {name: "route contains production runtime", route: filepath.Dir(prodRuntime), wantErr: true},
333+ {name: "unrelated route", route: filepath.Join(t.TempDir(), "mux.sock")},
334+ }
335+ for _, tt := range tests {
336+ t.Run(tt.name, func(t *testing.T) {
337+ root := t.TempDir()
338+ writeConfig(t, root, registrationDocument(map[string]any{"route": tt.route}), 0o600)
339+ layout, err := ResolveLayout(root, []string{prodRuntime, prodState})
340+ require.NoError(t, err)
341+ _, err = Load(layout)
342+ if !tt.wantErr {
343+ require.NoError(t, err)
344+ return
345+ }
346+ require.Error(t, err)
347+ require.Contains(t, err.Error(), "overlaps production path")
348+ })
349+ }
350+}
351+
352+func TestLayoutVerifyCreatedDetectsSymlinkedComponent(t *testing.T) {
353+ root := filepath.Join(t.TempDir(), "sandbox")
354+ layout, err := ResolveLayout(root, nil)
355+ require.NoError(t, err)
356+
357+ // Nothing is created yet, so the walk ends at the first missing component.
358+ require.NoError(t, layout.VerifyCreated())
359+
360+ require.NoError(t, os.MkdirAll(root, 0o700))
361+ require.NoError(t, os.Symlink(t.TempDir(), layout.State))
362+ require.ErrorContains(t, layout.VerifyCreated(), "symlink")
363+
364+ // Once the swap is removed, every created component is symlink-free again.
365+ require.NoError(t, os.Remove(layout.State))
366+ require.NoError(t, layout.VerifyCreated())
367+}
368+
369+// registrationsDocument renders one document carrying the supplied registration
370+// entries in order.
371+func registrationsDocument(entries ...map[string]any) map[string]any {
372+ list := make([]any, 0, len(entries))
373+ for _, entry := range entries {
374+ list = append(list, entry)
375+ }
376+ return map[string]any{"marker": Marker, "registrations": list}
377+}
378+
379+// registrationEntry renders one registration entry with the supplied overrides.
380+func registrationEntry(overrides map[string]any) map[string]any {
381+ return registrationDocument(overrides)["registrations"].([]any)[0].(map[string]any)
382+}
383+
384+// TestLoadEnforcesOneRoutePerPooledIdentity pins the pooling invariant: the
385+// broker pool keys one physical transport by the exact authenticated identity
386+// plus policy pair and deliberately ignores the opaque route address, so two
387+// registrations that share the pair must resolve to the same route. An alias on
388+// an identical route is accepted; a second route for the same pair is refused,
389+// because the pool would otherwise dial one of the two addresses and silently
390+// strand the other.
391+func TestLoadEnforcesOneRoutePerPooledIdentity(t *testing.T) {
392+ const (
393+ aliasOne = "alias@daemon:2222"
394+ aliasTwo = "alias@daemon:2223"
395+ otherID = "other@daemon:2224"
396+ routeOne = "/tmp/vev-brokerconfig-test/one.sock"
397+ routeTwo = "/tmp/vev-brokerconfig-test/two.sock"
398+ )
399+ withPolicy := func(entry map[string]any, key string, value any) map[string]any {
400+ entry["policy"].(map[string]any)[key] = value
401+ return entry
402+ }
403+ tests := []struct {
404+ name string
405+ entries []map[string]any
406+ wantErr string
407+ wantRoutes []string
408+ }{
409+ {
410+ name: "same identity and policy on a different route is refused",
411+ entries: []map[string]any{
412+ registrationEntry(map[string]any{"endpoint": aliasOne, "route": routeOne}),
413+ registrationEntry(map[string]any{"endpoint": aliasTwo, "route": routeTwo}),
414+ },
415+ wantErr: "different route",
416+ },
417+ {
418+ name: "same identity and policy on the identical route is an alias",
419+ entries: []map[string]any{
420+ registrationEntry(map[string]any{"endpoint": aliasOne, "route": routeOne}),
421+ registrationEntry(map[string]any{"endpoint": aliasTwo, "route": routeOne}),
422+ },
423+ wantRoutes: []string{aliasOne, aliasTwo},
424+ },
425+ {
426+ name: "distinct identities on different routes are independent",
427+ entries: []map[string]any{
428+ registrationEntry(map[string]any{"endpoint": aliasOne, "route": routeOne}),
429+ registrationEntry(map[string]any{"endpoint": otherID, "route": routeTwo, "identity": "other-daemon"}),
430+ },
431+ wantRoutes: []string{aliasOne, otherID},
432+ },
433+ {
434+ name: "same identity with a distinct policy on a different route is independent",
435+ entries: []map[string]any{
436+ registrationEntry(map[string]any{"endpoint": aliasOne, "route": routeOne}),
437+ withPolicy(registrationEntry(map[string]any{"endpoint": aliasTwo, "route": routeTwo}), "transport", "other-transport"),
438+ },
439+ wantRoutes: []string{aliasOne, aliasTwo},
440+ },
441+ }
442+ for _, tt := range tests {
443+ t.Run(tt.name, func(t *testing.T) {
444+ root := t.TempDir()
445+ writeConfig(t, root, registrationsDocument(tt.entries...), 0o600)
446+ layout, err := ResolveLayout(root, nil)
447+ require.NoError(t, err)
448+ config, err := Load(layout)
449+ if tt.wantErr != "" {
450+ require.Error(t, err)
451+ require.Contains(t, err.Error(), tt.wantErr)
452+ return
453+ }
454+ require.NoError(t, err)
455+ require.Equal(t, tt.wantRoutes, config.Endpoints())
456+ })
457+ }
458+}
459+
460+// TestLoadAliasesShareOneRoute proves an accepted alias resolves both endpoints
461+// to the identical opaque address and that the shared local route is still
462+// reported exactly once, so a remote-side helper is never handed a duplicate of
463+// the same carriage.
464+func TestLoadAliasesShareOneRoute(t *testing.T) {
465+ const (
466+ first = "alias@daemon:2222"
467+ second = "alias@daemon:2223"
468+ route = "/tmp/vev-brokerconfig-test/shared.sock"
469+ )
470+ root := t.TempDir()
471+ writeConfig(t, root, registrationsDocument(
472+ registrationEntry(map[string]any{"endpoint": first, "route": route}),
473+ registrationEntry(map[string]any{"endpoint": second, "route": route}),
474+ ), 0o600)
475+ layout, err := ResolveLayout(root, nil)
476+ require.NoError(t, err)
477+ config, err := Load(layout)
478+ require.NoError(t, err)
479+
480+ local, err := config.LocalMuxRoute()
481+ require.NoError(t, err)
482+ require.Equal(t, route, local.Path())
483+
484+ for _, endpoint := range []string{first, second} {
485+ registration := testRegistration()
486+ registration.Endpoint = endpoint
487+ resolved, err := config.Resolver().Resolve(context.Background(), ports.BrokerOpenStreamRequest{
488+ Purpose: ports.BrokerStreamControl,
489+ Endpoint: endpoint,
490+ Registration: registration,
491+ Policy: testPolicy(),
492+ })
493+ require.NoError(t, err)
494+ require.Equal(t, local.Address(), resolved.Address)
495+ }
496+}
497+
498+// documentWithPolicy renders a valid document with one policy field overridden.
499+func documentWithPolicy(t *testing.T, key string, value any) map[string]any {
500+ t.Helper()
501+ document := registrationDocument(nil)
502+ entry := document["registrations"].([]any)[0].(map[string]any)
503+ entry["policy"].(map[string]any)[key] = value
504+ return document
505+}
506+
507+// writeRawConfig writes one raw config file with owner-only permissions.
508+func writeRawConfig(t *testing.T, root, content string) {
509+ t.Helper()
510+ require.NoError(t, os.WriteFile(filepath.Join(root, ConfigFileName), []byte(content), 0o600))
511+}
512+
513+// testRegistration is the exact registration the fixture document provisions.
514+func testRegistration() domain.RemoteRegistration {
515+ var registration domain.RemoteRegistration
516+ registration.Endpoint = testEndpoint
517+ for i := range registration.Incarnation {
518+ registration.Incarnation[i] = byte(i + 1)
519+ }
520+ registration.Generation = 1
521+ return registration
522+}
523+
524+func TestResolverRefusals(t *testing.T) {
525+ root := t.TempDir()
526+ writeConfig(t, root, registrationDocument(nil), 0o600)
527+ layout, err := ResolveLayout(root, nil)
528+ require.NoError(t, err)
529+ config, err := Load(layout)
530+ require.NoError(t, err)
531+ resolver := config.Resolver()
532+
533+ base := ports.BrokerOpenStreamRequest{
534+ Purpose: ports.BrokerStreamControl,
535+ Endpoint: testEndpoint,
536+ Registration: testRegistration(),
537+ Policy: testPolicy(),
538+ }
539+
540+ t.Run("unknown endpoint", func(t *testing.T) {
541+ request := base
542+ request.Endpoint = "other@daemon"
543+ _, err := resolver.Resolve(context.Background(), request)
544+ var typed ports.BrokerError
545+ require.ErrorAs(t, err, &typed)
546+ require.Equal(t, ports.BrokerErrorUnavailable, typed.Code)
547+ require.ErrorIs(t, err, errUnknownEndpoint)
548+ })
549+ t.Run("local request", func(t *testing.T) {
550+ request := base
551+ request.Local = true
552+ _, err := resolver.Resolve(context.Background(), request)
553+ var typed ports.BrokerError
554+ require.ErrorAs(t, err, &typed)
555+ require.Equal(t, ports.BrokerErrorUnavailable, typed.Code)
556+ require.ErrorIs(t, err, errNoLocalRoute)
557+ })
558+ t.Run("stale registration", func(t *testing.T) {
559+ request := base
560+ request.Registration.Generation = 2
561+ _, err := resolver.Resolve(context.Background(), request)
562+ var typed ports.BrokerError
563+ require.ErrorAs(t, err, &typed)
564+ require.Equal(t, ports.BrokerErrorUnavailable, typed.Code)
565+ require.ErrorIs(t, err, errStaleRegistration)
566+ })
567+ t.Run("conflicting policy", func(t *testing.T) {
568+ request := base
569+ request.Policy.Transport = "other-transport"
570+ _, err := resolver.Resolve(context.Background(), request)
571+ var typed ports.BrokerError
572+ require.ErrorAs(t, err, &typed)
573+ require.Equal(t, ports.BrokerErrorConflictingPolicy, typed.Code)
574+ })
575+ t.Run("cancelled context", func(t *testing.T) {
576+ ctx, cancel := context.WithCancel(context.Background())
577+ cancel()
578+ _, err := resolver.Resolve(ctx, base)
579+ require.ErrorIs(t, err, context.Canceled)
580+ })
581+}
582+
583+func TestDecodeStrictRejectsNesting(t *testing.T) {
584+ root := t.TempDir()
585+ nested := strings.Repeat("[", 130) + strings.Repeat("]", 130)
586+ writeRawConfig(t, root, `{"marker":"`+Marker+`","registrations":`+nested+`}`)
587+ layout, err := ResolveLayout(root, nil)
588+ require.NoError(t, err)
589+ _, err = Load(layout)
590+ require.Error(t, err)
591+ require.Contains(t, err.Error(), "nesting")
592+}
593+
594+func TestResolveLayoutDoesNotCreatePaths(t *testing.T) {
595+ root := filepath.Join(t.TempDir(), "does-not-exist")
596+ layout, err := ResolveLayout(root, nil)
597+ require.NoError(t, err)
598+ require.Equal(t, filepath.Join(root, StateDirName), layout.State)
599+ require.Equal(t, filepath.Join(root, RuntimeDirName, SpawnDirName), layout.Spawn)
600+ _, statErr := os.Lstat(root)
601+ require.ErrorIs(t, statErr, os.ErrNotExist)
602+}
603+
604+func TestLoadIdleGrace(t *testing.T) {
605+ tests := []struct {
606+ name string
607+ document map[string]any
608+ want time.Duration
609+ wantSet bool
610+ wantErr string
611+ }{
612+ {
613+ name: "provisioned grace is parsed",
614+ document: map[string]any{"marker": Marker, "registrations": []any{}, "idleGrace": "90s"},
615+ want: 90 * time.Second,
616+ wantSet: true,
617+ },
618+ {name: "absent grace provisions nothing", document: map[string]any{"marker": Marker, "registrations": []any{}}},
619+ {
620+ name: "invalid duration is refused",
621+ document: map[string]any{"marker": Marker, "registrations": []any{}, "idleGrace": "soon"},
622+ wantErr: "not a duration",
623+ },
624+ {
625+ name: "non-positive grace is refused",
626+ document: map[string]any{"marker": Marker, "registrations": []any{}, "idleGrace": "0s"},
627+ wantErr: "must be positive",
628+ },
629+ {
630+ name: "oversized grace is refused",
631+ document: map[string]any{"marker": Marker, "registrations": []any{}, "idleGrace": (MaxIdleGrace + time.Second).String()},
632+ wantErr: "must be positive and at most",
633+ },
634+ }
635+ for _, tt := range tests {
636+ t.Run(tt.name, func(t *testing.T) {
637+ root := t.TempDir()
638+ writeConfig(t, root, tt.document, 0o600)
639+ layout, err := ResolveLayout(root, nil)
640+ require.NoError(t, err)
641+ config, err := Load(layout)
642+ if tt.wantErr != "" {
643+ require.Error(t, err)
644+ require.Contains(t, err.Error(), tt.wantErr)
645+ return
646+ }
647+ require.NoError(t, err)
648+ grace, set := config.IdleGrace()
649+ require.Equal(t, tt.wantSet, set)
650+ require.Equal(t, tt.want, grace)
651+ })
652+ }
653+}
A internal/adapters/brokerconfig/resolver.go +52 -0
@@ -0,0 +1,52 @@
1+package brokerconfig
2+
3+import (
4+ "context"
5+ "errors"
6+
7+ "github.com/bnema/vev/internal/ports"
8+)
9+
10+// Resolver is the immutable ports.BrokerEndpointResolver over one parsed
11+// offline configuration. It holds no mutable state and performs no I/O: every
12+// Resolve call fences the request against the provisioned registration and
13+// returns the configured identity, policy, and opaque route address. A request
14+// can therefore never select an address, identity, or policy of its own.
15+type Resolver struct {
16+ byEndpoint map[string]Registration
17+}
18+
19+var _ ports.BrokerEndpointResolver = (*Resolver)(nil)
20+
21+// Resolve fences one open-stream request against the immutable configuration.
22+//
23+// The request must name a configured endpoint, carry that endpoint's exact
24+// registration (endpoint, incarnation, and generation), and request a policy
25+// compatible with the provisioned policy. A local request is refused because
26+// the offline sandbox provisions remote routes only. Unknown, stale,
27+// or conflicting requests are refused with a typed broker error and no dial is
28+// attempted; a cancelled context is reported as such so the pool classifies it
29+// as cancellation rather than unavailability.
30+func (r *Resolver) Resolve(ctx context.Context, request ports.BrokerOpenStreamRequest) (ports.BrokerResolvedEndpoint, error) {
31+ if err := ctx.Err(); err != nil {
32+ return ports.BrokerResolvedEndpoint{}, err
33+ }
34+ if request.Local {
35+ return ports.BrokerResolvedEndpoint{}, ports.BrokerError{Code: ports.BrokerErrorUnavailable, Cause: errNoLocalRoute}
36+ }
37+ provisioned, ok := r.byEndpoint[request.Endpoint]
38+ if !ok {
39+ return ports.BrokerResolvedEndpoint{}, ports.BrokerError{Code: ports.BrokerErrorUnavailable, Cause: errUnknownEndpoint}
40+ }
41+ if !request.Registration.Equal(provisioned.Registration) {
42+ return ports.BrokerResolvedEndpoint{}, ports.BrokerError{Code: ports.BrokerErrorUnavailable, Cause: errStaleRegistration}
43+ }
44+ if !provisioned.Policy.Compatible(request.Policy) {
45+ return ports.BrokerResolvedEndpoint{}, ports.BrokerError{Code: ports.BrokerErrorConflictingPolicy, Cause: errors.New("brokerconfig: request policy conflicts with the provisioned policy")}
46+ }
47+ resolved := ports.BrokerResolvedEndpoint{Identity: provisioned.Identity, Policy: provisioned.Policy, Address: provisioned.Route.Address()}
48+ if err := resolved.Validate(); err != nil {
49+ return ports.BrokerResolvedEndpoint{}, ports.BrokerError{Code: ports.BrokerErrorIncompatible, Cause: err}
50+ }
51+ return resolved, nil
52+}
A internal/adapters/brokerconfig/route.go +364 -0
@@ -0,0 +1,364 @@
1+package brokerconfig
2+
3+import (
4+ "bytes"
5+ "crypto/sha256"
6+ "encoding/hex"
7+ "encoding/json"
8+ "errors"
9+ "fmt"
10+ "io"
11+ "path/filepath"
12+ "strings"
13+ "time"
14+ "unicode"
15+ "unicode/utf8"
16+
17+ "github.com/bnema/vev/internal/domain"
18+)
19+
20+// Bounded route variants (Plan 001 P3.4 slice E).
21+//
22+// A registration's route is the transport the offline broker dials for that
23+// endpoint. It is an explicit, bounded tagged union, never a caller-supplied
24+// address: the immutable configuration alone decides whether the broker reaches
25+// the daemon-mux carriage over a local Unix socket, an SSH stdio child, or an
26+// SSH-authenticated QUIC bootstrap. The route carries no identity, policy, or
27+// secret; identity and policy stay authoritative on the registration, and the
28+// one-time QUIC credentials are minted fresh per physical connection and never
29+// enter an address, a log, or the configuration.
30+//
31+// The route's opaque pool address is a digest of the route's own bounded
32+// contents, so two distinct routes can never collide and no route detail (an
33+// SSH target, a path, or an argv word) is exposed through the address the pool
34+// carries.
35+type RouteKind string
36+
37+const (
38+ // RouteUnix dials one absolute private daemon-mux Unix socket directly.
39+ RouteUnix RouteKind = "unix"
40+ // RouteSSHStdio starts one explicit remote mux helper over a local ssh
41+ // child and carries the daemonmux conversation over its stdio.
42+ RouteSSHStdio RouteKind = "ssh-stdio"
43+ // RouteSSHQUIC runs one SSH-authenticated QUIC bootstrap on the remote
44+ // host and pins the freshly minted ephemeral certificate before dialing.
45+ RouteSSHQUIC RouteKind = "ssh-quic"
46+)
47+
48+// Bounded route limits. They are explicit so a hostile or accidental
49+// configuration can never grow a route, an argv, or a trust input without
50+// limit, and so two distinct routes are always distinguishable inside the
51+// address digest.
52+const (
53+ // MaxRouteTargetBytes bounds one ssh target.
54+ MaxRouteTargetBytes = 256
55+ // MaxRouteArgvWords bounds the remote argv word count.
56+ MaxRouteArgvWords = 32
57+ // MaxRouteArgvBytes bounds the total serialized remote argv.
58+ MaxRouteArgvBytes = 4096
59+ // MaxRouteWordBytes bounds one remote argv word.
60+ MaxRouteWordBytes = 1024
61+ // MaxTrustPathBytes bounds one trust-input path.
62+ MaxTrustPathBytes = 1024
63+ // MaxConnectTimeout bounds a provisioned ssh connect timeout.
64+ MaxConnectTimeout = 5 * time.Minute
65+
66+ // MaxRouteAddressBytes bounds the opaque pool address a route exposes.
67+ MaxRouteAddressBytes = 64
68+)
69+
70+// TrustInputs carries the explicit OpenSSH inputs a provisioned ssh route may
71+// forward. They can only narrow or redirect verification; host-key checking is
72+// never disabled, authentication is never bypassed, and no PTY is ever
73+// requested. Every field is optional.
74+type TrustInputs struct {
75+ // knownHostsFile, when set, names the absolute known-hosts file ssh verifies
76+ // against. Verification still happens: an unknown or mismatched key fails.
77+ knownHostsFile string
78+ // connectTimeout, when positive, bounds ssh's own connection attempt.
79+ connectTimeout time.Duration
80+ // set records whether the trust object was present, so an empty object and
81+ // an absent object are distinguishable in the route digest.
82+ set bool
83+}
84+
85+// Route is one immutable, validated daemonmux route.
86+type Route struct {
87+ kind RouteKind
88+ path string
89+ target string
90+ argv []string
91+ trust TrustInputs
92+ // address is the bounded opaque token the pool carries for this route. It
93+ // is a digest of the route contents, so it leaks no route detail and can
94+ // never collide with a distinct route.
95+ address string
96+ // local is true for a Unix route that this process can dial directly, which
97+ // is the route a remote-side mux helper bridges to.
98+ local bool
99+}
100+
101+// Kind returns the route variant. The zero Route reports an empty kind.
102+func (r Route) Kind() RouteKind { return r.kind }
103+
104+// Path returns the absolute private Unix mux socket for a Unix route.
105+func (r Route) Path() string { return r.path }
106+
107+// Target returns the ssh target for an ssh route.
108+func (r Route) Target() string { return r.target }
109+
110+// Argv returns a copy of the explicit remote argv for an ssh route.
111+func (r Route) Argv() []string { return append([]string(nil), r.argv...) }
112+
113+// KnownHostsFile returns the provisioned known-hosts path, if any.
114+func (r Route) KnownHostsFile() string { return r.trust.knownHostsFile }
115+
116+// ConnectTimeout returns the provisioned ssh connect timeout, if any.
117+func (r Route) ConnectTimeout() time.Duration { return r.trust.connectTimeout }
118+
119+// Address returns the bounded opaque pool address for this route. It never
120+// contains a secret, an argv word, or a trust path.
121+func (r Route) Address() string { return r.address }
122+
123+// IsLocal reports whether the route dials a Unix socket this process can reach
124+// directly, which is the route a remote-side mux helper bridges to.
125+func (r Route) IsLocal() bool { return r.local }
126+
127+// Validate re-checks one already-parsed route. Parsed routes are validated
128+// once, so this is the seam a caller uses to fence a zero or hand-built value.
129+func (r Route) Validate() error {
130+ switch r.kind {
131+ case RouteUnix:
132+ if err := validateRoute(r.path); err != nil {
133+ return err
134+ }
135+ return nil
136+ case RouteSSHStdio, RouteSSHQUIC:
137+ return r.validateSSH()
138+ default:
139+ return fmt.Errorf("route kind %q is invalid", r.kind)
140+ }
141+}
142+
143+func (r Route) validateSSH() error {
144+ if err := validateSSHTarget(r.target); err != nil {
145+ return err
146+ }
147+ if err := validateArgv(r.argv); err != nil {
148+ return err
149+ }
150+ if r.trust.knownHostsFile != "" {
151+ if err := validateTrustPath(r.trust.knownHostsFile); err != nil {
152+ return err
153+ }
154+ }
155+ if r.trust.connectTimeout < 0 || r.trust.connectTimeout > MaxConnectTimeout {
156+ return fmt.Errorf("trust connectTimeout must be between 0 and %s", MaxConnectTimeout)
157+ }
158+ return nil
159+}
160+
161+// validateSSHTarget bounds and sanitizes one ssh target. It reuses the domain
162+// host-target rule (no whitespace or control characters) and additionally
163+// bounds the length and refuses a leading dash, so a target can never be
164+// mistaken for an ssh option even before the option terminator.
165+func validateSSHTarget(target string) error {
166+ if err := domain.ValidateRemoteHostTarget(target); err != nil {
167+ return err
168+ }
169+ if len(target) > MaxRouteTargetBytes {
170+ return fmt.Errorf("ssh target exceeds %d bytes", MaxRouteTargetBytes)
171+ }
172+ if strings.HasPrefix(target, "-") {
173+ return errors.New("ssh target must not start with a dash")
174+ }
175+ return nil
176+}
177+
178+// validateArgv bounds the explicit remote argv. Every word must be non-empty,
179+// valid UTF-8, and free of NUL and control characters; the executable is the
180+// first word. The remote argv is passed to ssh as one POSIX-quoted string, so
181+// spaces inside a word are safe and are not treated as separators.
182+func validateArgv(argv []string) error {
183+ if len(argv) == 0 {
184+ return errors.New("ssh route requires a non-empty argv")
185+ }
186+ if len(argv) > MaxRouteArgvWords {
187+ return fmt.Errorf("ssh route argv exceeds %d words", MaxRouteArgvWords)
188+ }
189+ total := 0
190+ for i, word := range argv {
191+ if word == "" {
192+ return fmt.Errorf("ssh route argv[%d] is empty", i)
193+ }
194+ if len(word) > MaxRouteWordBytes {
195+ return fmt.Errorf("ssh route argv[%d] exceeds %d bytes", i, MaxRouteWordBytes)
196+ }
197+ if !utf8.ValidString(word) {
198+ return fmt.Errorf("ssh route argv[%d] is not valid UTF-8", i)
199+ }
200+ for _, r := range word {
201+ if r == 0 || unicode.IsControl(r) {
202+ return fmt.Errorf("ssh route argv[%d] contains a control character", i)
203+ }
204+ }
205+ total += len(word)
206+ }
207+ if total > MaxRouteArgvBytes {
208+ return fmt.Errorf("ssh route argv exceeds %d bytes", MaxRouteArgvBytes)
209+ }
210+ return nil
211+}
212+
213+// validateTrustPath bounds one trust-input path. It must be absolute and clean
214+// and free of control characters, so it can never be reshaped by a later step
215+// or inject an ssh option.
216+func validateTrustPath(path string) error {
217+ if len(path) > MaxTrustPathBytes {
218+ return fmt.Errorf("trust path exceeds %d bytes", MaxTrustPathBytes)
219+ }
220+ if !utf8.ValidString(path) {
221+ return errors.New("trust path is not valid UTF-8")
222+ }
223+ if !filepath.IsAbs(path) {
224+ return fmt.Errorf("trust path %q is not absolute", path)
225+ }
226+ if filepath.Clean(path) != path {
227+ return fmt.Errorf("trust path %q is not clean", path)
228+ }
229+ for _, r := range path {
230+ if r == 0 || unicode.IsControl(r) || unicode.IsSpace(r) {
231+ return fmt.Errorf("trust path %q contains whitespace or control characters", path)
232+ }
233+ }
234+ return nil
235+}
236+
237+// routeDocument is the strict on-disk shape of a route object. Unknown fields
238+// are refused by the decoder, so a route cannot smuggle additional behavior.
239+type routeDocument struct {
240+ Kind string `json:"kind"`
241+ Path string `json:"path,omitempty"`
242+ Target string `json:"target,omitempty"`
243+ Argv []string `json:"argv,omitempty"`
244+ Trust *trustDocument `json:"trust,omitempty"`
245+}
246+
247+type trustDocument struct {
248+ KnownHostsFile string `json:"knownHostsFile,omitempty"`
249+ ConnectTimeout string `json:"connectTimeout,omitempty"`
250+}
251+
252+// parseRoute strictly parses one route. A JSON string is the original Unix
253+// route form (an absolute private mux path); an object carries an explicit
254+// kind. Unknown kinds, unknown fields, trailing JSON, and a kind/field mismatch
255+// are refused.
256+func parseRoute(raw json.RawMessage) (Route, error) {
257+ trimmed := bytes.TrimSpace(raw)
258+ if len(trimmed) == 0 {
259+ return Route{}, errors.New("route is empty")
260+ }
261+ if trimmed[0] == '"' {
262+ var path string
263+ if err := json.Unmarshal(trimmed, &path); err != nil {
264+ return Route{}, fmt.Errorf("route: %w", err)
265+ }
266+ return newUnixRoute(path)
267+ }
268+ if trimmed[0] != '{' {
269+ return Route{}, errors.New("route must be a path string or an object")
270+ }
271+ var document routeDocument
272+ decoder := json.NewDecoder(bytes.NewReader(trimmed))
273+ decoder.DisallowUnknownFields()
274+ if err := decoder.Decode(&document); err != nil {
275+ return Route{}, fmt.Errorf("route: %w", err)
276+ }
277+ if _, err := decoder.Token(); err != io.EOF {
278+ return Route{}, errors.New("route: trailing JSON")
279+ }
280+ switch RouteKind(document.Kind) {
281+ case RouteUnix:
282+ if document.Target != "" || len(document.Argv) != 0 || document.Trust != nil {
283+ return Route{}, errors.New("unix route must not carry ssh fields")
284+ }
285+ return newUnixRoute(document.Path)
286+ case RouteSSHStdio, RouteSSHQUIC:
287+ trust, err := parseTrust(document.Trust)
288+ if err != nil {
289+ return Route{}, err
290+ }
291+ return newSSHRoute(RouteKind(document.Kind), document.Target, document.Argv, trust)
292+ case "":
293+ return Route{}, errors.New("route object requires a kind")
294+ default:
295+ return Route{}, fmt.Errorf("route kind %q is invalid", document.Kind)
296+ }
297+}
298+
299+func parseTrust(document *trustDocument) (TrustInputs, error) {
300+ if document == nil {
301+ return TrustInputs{}, nil
302+ }
303+ trust := TrustInputs{knownHostsFile: document.KnownHostsFile, set: true}
304+ if document.ConnectTimeout != "" {
305+ timeout, err := time.ParseDuration(document.ConnectTimeout)
306+ if err != nil {
307+ return TrustInputs{}, fmt.Errorf("trust connectTimeout %q is not a duration", document.ConnectTimeout)
308+ }
309+ if timeout <= 0 {
310+ return TrustInputs{}, errors.New("trust connectTimeout must be positive")
311+ }
312+ trust.connectTimeout = timeout
313+ }
314+ return trust, nil
315+}
316+
317+func newUnixRoute(path string) (Route, error) {
318+ route := Route{kind: RouteUnix, path: path, local: true}
319+ if err := route.Validate(); err != nil {
320+ return Route{}, err
321+ }
322+ route.address = route.digest()
323+ return route, nil
324+}
325+
326+func newSSHRoute(kind RouteKind, target string, argv []string, trust TrustInputs) (Route, error) {
327+ route := Route{
328+ kind: kind,
329+ target: target,
330+ argv: append([]string(nil), argv...),
331+ trust: trust,
332+ }
333+ if err := route.Validate(); err != nil {
334+ return Route{}, err
335+ }
336+ route.address = route.digest()
337+ return route, nil
338+}
339+
340+// digest computes the bounded opaque pool address. The canonical encoding
341+// separates every field with a NUL and carries the kind first, so distinct
342+// routes always produce distinct digests.
343+func (r Route) digest() string {
344+ var builder strings.Builder
345+ builder.WriteString(string(r.kind))
346+ builder.WriteByte(0)
347+ builder.WriteString(r.path)
348+ builder.WriteByte(0)
349+ builder.WriteString(r.target)
350+ builder.WriteByte(0)
351+ for _, word := range r.argv {
352+ builder.WriteString(word)
353+ builder.WriteByte(0)
354+ }
355+ builder.WriteByte(0)
356+ builder.WriteString(r.trust.knownHostsFile)
357+ builder.WriteByte(0)
358+ fmt.Fprintf(&builder, "%d", r.trust.connectTimeout)
359+ if r.trust.set {
360+ builder.WriteByte(1)
361+ }
362+ sum := sha256.Sum256([]byte(builder.String()))
363+ return "offline-route-" + hex.EncodeToString(sum[:16])
364+}
A internal/adapters/brokerconfig/route_test.go +208 -0
@@ -0,0 +1,208 @@
1+package brokerconfig
2+
3+import (
4+ "encoding/json"
5+ "strings"
6+ "testing"
7+ "time"
8+
9+ "github.com/stretchr/testify/require"
10+)
11+
12+// TestParseRouteVariants pins the explicit bounded route union: a bare string
13+// is the original Unix path form, an object carries an explicit kind, and every
14+// kind/field mismatch or bound violation is refused.
15+func TestParseRouteVariants(t *testing.T) {
16+ t.Parallel()
17+
18+ tests := []struct {
19+ name string
20+ raw string
21+ wantErr string
22+ check func(t *testing.T, route Route)
23+ }{
24+ {
25+ name: "bare string is a Unix route",
26+ raw: `"/tmp/mux.sock"`,
27+ check: func(t *testing.T, route Route) {
28+ require.Equal(t, RouteUnix, route.Kind())
29+ require.Equal(t, "/tmp/mux.sock", route.Path())
30+ require.True(t, route.IsLocal())
31+ },
32+ },
33+ {
34+ name: "explicit unix route",
35+ raw: `{"kind":"unix","path":"/tmp/mux.sock"}`,
36+ check: func(t *testing.T, route Route) {
37+ require.Equal(t, RouteUnix, route.Kind())
38+ require.Equal(t, "/tmp/mux.sock", route.Path())
39+ },
40+ },
41+ {
42+ name: "ssh stdio route with trust inputs",
43+ raw: `{"kind":"ssh-stdio","target":"user@host:2222","argv":["vev","_broker-mux-stdio","--offline-root","/srv/vev"],` +
44+ `"trust":{"knownHostsFile":"/etc/ssh/known_hosts","connectTimeout":"30s"}}`,
45+ check: func(t *testing.T, route Route) {
46+ require.Equal(t, RouteSSHStdio, route.Kind())
47+ require.Equal(t, "user@host:2222", route.Target())
48+ require.Equal(t, []string{"vev", "_broker-mux-stdio", "--offline-root", "/srv/vev"}, route.Argv())
49+ require.Equal(t, "/etc/ssh/known_hosts", route.KnownHostsFile())
50+ require.Equal(t, 30*time.Second, route.ConnectTimeout())
51+ require.False(t, route.IsLocal())
52+ },
53+ },
54+ {
55+ name: "ssh quic route without trust",
56+ raw: `{"kind":"ssh-quic","target":"host","argv":["vev","_broker-mux-quic-bootstrap","--offline-root","/srv/vev"]}`,
57+ check: func(t *testing.T, route Route) {
58+ require.Equal(t, RouteSSHQUIC, route.Kind())
59+ require.Equal(t, "host", route.Target())
60+ require.Empty(t, route.KnownHostsFile())
61+ require.Zero(t, route.ConnectTimeout())
62+ },
63+ },
64+ {name: "empty route", raw: ``, wantErr: "empty"},
65+ {name: "null route", raw: `null`, wantErr: "path string or an object"},
66+ {name: "number route", raw: `42`, wantErr: "path string or an object"},
67+ {name: "unknown kind", raw: `{"kind":"carrier-pigeon","target":"h","argv":["x"]}`, wantErr: "invalid"},
68+ {name: "missing kind", raw: `{"path":"/tmp/mux.sock"}`, wantErr: "requires a kind"},
69+ {name: "unknown field", raw: `{"kind":"unix","path":"/tmp/mux.sock","extra":1}`, wantErr: "unknown field"},
70+ {name: "trailing json", raw: `{"kind":"unix","path":"/tmp/mux.sock"} {}`, wantErr: "trailing"},
71+ {name: "unix carries ssh fields", raw: `{"kind":"unix","path":"/tmp/mux.sock","target":"h"}`, wantErr: "must not carry ssh"},
72+ {name: "ssh without argv", raw: `{"kind":"ssh-stdio","target":"h"}`, wantErr: "argv"},
73+ {name: "ssh empty argv word", raw: `{"kind":"ssh-stdio","target":"h","argv":["vev",""]}`, wantErr: "empty"},
74+ {name: "ssh word control char", raw: `{"kind":"ssh-stdio","target":"h","argv":["vev","a\nb"]}`, wantErr: "control"},
75+ {name: "ssh target whitespace", raw: `{"kind":"ssh-stdio","target":"a b","argv":["vev"]}`, wantErr: "whitespace"},
76+ {name: "ssh target leading dash", raw: `{"kind":"ssh-stdio","target":"-oProxyCommand=x","argv":["vev"]}`, wantErr: "dash"},
77+ {name: "ssh target too long", raw: `{"kind":"ssh-stdio","target":"` + strings.Repeat("a", MaxRouteTargetBytes+1) + `","argv":["vev"]}`, wantErr: "exceeds"},
78+ {name: "too many argv words", raw: `{"kind":"ssh-stdio","target":"h","argv":[` + strings.Repeat(`"w",`, MaxRouteArgvWords) + `"w"]}`, wantErr: "words"},
79+ {name: "relative trust path", raw: `{"kind":"ssh-stdio","target":"h","argv":["vev"],"trust":{"knownHostsFile":"known_hosts"}}`, wantErr: "not absolute"},
80+ {name: "unclean trust path", raw: `{"kind":"ssh-stdio","target":"h","argv":["vev"],"trust":{"knownHostsFile":"/etc/ssh/../known_hosts"}}`, wantErr: "not clean"},
81+ {name: "space in trust path", raw: `{"kind":"ssh-stdio","target":"h","argv":["vev"],"trust":{"knownHostsFile":"/etc/ssh/known hosts"}}`, wantErr: "whitespace"},
82+ {name: "bad timeout duration", raw: `{"kind":"ssh-stdio","target":"h","argv":["vev"],"trust":{"connectTimeout":"soon"}}`, wantErr: "not a duration"},
83+ {name: "negative timeout", raw: `{"kind":"ssh-stdio","target":"h","argv":["vev"],"trust":{"connectTimeout":"-1s"}}`, wantErr: "positive"},
84+ {name: "excessive timeout", raw: `{"kind":"ssh-stdio","target":"h","argv":["vev"],"trust":{"connectTimeout":"1h"}}`, wantErr: "between"},
85+ }
86+ for _, tt := range tests {
87+ t.Run(tt.name, func(t *testing.T) {
88+ route, err := parseRoute(json.RawMessage(tt.raw))
89+ if tt.wantErr != "" {
90+ require.Error(t, err)
91+ require.Contains(t, err.Error(), tt.wantErr)
92+ return
93+ }
94+ require.NoError(t, err)
95+ require.NoError(t, route.Validate())
96+ if tt.check != nil {
97+ tt.check(t, route)
98+ }
99+ })
100+ }
101+}
102+
103+// TestRouteAddressIsOpaqueAndDeterministic proves the pool address leaks no
104+// route detail, is bounded, is stable across parses, and distinguishes distinct
105+// routes.
106+func TestRouteAddressIsOpaqueAndDeterministic(t *testing.T) {
107+ t.Parallel()
108+
109+ unix, err := parseRoute(json.RawMessage(`{"kind":"unix","path":"/tmp/secret-dir/mux.sock"}`))
110+ require.NoError(t, err)
111+ stdio, err := parseRoute(json.RawMessage(`{"kind":"ssh-stdio","target":"user@secret-host","argv":["vev","_broker-mux-stdio","--offline-root","/srv/secret"]}`))
112+ require.NoError(t, err)
113+ quic, err := parseRoute(json.RawMessage(`{"kind":"ssh-quic","target":"user@secret-host","argv":["vev","_broker-mux-quic-bootstrap","--offline-root","/srv/secret"]}`))
114+ require.NoError(t, err)
115+
116+ require.NotEmpty(t, unix.Address())
117+ require.LessOrEqual(t, len(unix.Address()), MaxRouteAddressBytes)
118+ require.Contains(t, unix.Address(), "offline-route-")
119+
120+ for _, secret := range []string{"secret-host", "secret-dir", "/srv/secret", "_broker-mux", "mux.sock"} {
121+ for _, route := range []Route{unix, stdio, quic} {
122+ require.NotContains(t, route.Address(), secret, "address must not leak route detail")
123+ }
124+ }
125+ require.NotEqual(t, unix.Address(), stdio.Address())
126+ require.NotEqual(t, stdio.Address(), quic.Address())
127+
128+ again, err := parseRoute(json.RawMessage(`{"kind":"ssh-stdio","target":"user@secret-host","argv":["vev","_broker-mux-stdio","--offline-root","/srv/secret"]}`))
129+ require.NoError(t, err)
130+ require.Equal(t, stdio.Address(), again.Address(), "address must be stable across parses")
131+
132+ // A different trust input is a different route and therefore a different
133+ // address.
134+ trusted, err := parseRoute(json.RawMessage(`{"kind":"ssh-stdio","target":"user@secret-host","argv":["vev","_broker-mux-stdio","--offline-root","/srv/secret"],"trust":{"connectTimeout":"5s"}}`))
135+ require.NoError(t, err)
136+ require.NotEqual(t, stdio.Address(), trusted.Address())
137+}
138+
139+// TestConfigLocalMuxRouteRules proves a helper only ever bridges to exactly one
140+// provisioned Unix route.
141+func TestConfigLocalMuxRouteRules(t *testing.T) {
142+ t.Parallel()
143+
144+ root := t.TempDir()
145+ writeConfig(t, root, emptyRouteConfig(), 0o600)
146+ layout, err := ResolveLayout(root, nil)
147+ require.NoError(t, err)
148+ config, err := Load(layout)
149+ require.NoError(t, err)
150+ _, err = config.LocalMuxRoute()
151+ require.Error(t, err)
152+ require.Contains(t, err.Error(), "no Unix daemon-mux route")
153+
154+ root = t.TempDir()
155+ writeConfig(t, root, routeConfig(
156+ map[string]any{"kind": "unix", "path": "/tmp/one.sock"},
157+ map[string]any{"kind": "unix", "path": "/tmp/two.sock"},
158+ ), 0o600)
159+ layout, err = ResolveLayout(root, nil)
160+ require.NoError(t, err)
161+ config, err = Load(layout)
162+ require.NoError(t, err)
163+ _, err = config.LocalMuxRoute()
164+ require.Error(t, err)
165+ require.Contains(t, err.Error(), "more than one")
166+
167+ root = t.TempDir()
168+ writeConfig(t, root, routeConfig(map[string]any{"kind": "unix", "path": "/tmp/one.sock"}), 0o600)
169+ layout, err = ResolveLayout(root, nil)
170+ require.NoError(t, err)
171+ config, err = Load(layout)
172+ require.NoError(t, err)
173+ route, err := config.LocalMuxRoute()
174+ require.NoError(t, err)
175+ require.Equal(t, "/tmp/one.sock", route.Path())
176+}
177+
178+// emptyRouteConfig renders a marker-valid document with no registrations.
179+func emptyRouteConfig() map[string]any {
180+ return map[string]any{"marker": Marker, "registrations": []any{}}
181+}
182+
183+// routeConfig renders one marker-valid document whose registrations carry the
184+// supplied explicit route objects. Each registration gets a distinct endpoint
185+// and authenticated identity, so a document with several routes exercises the
186+// route rules rather than the one-route-per-pooled-identity rule.
187+func routeConfig(routes ...map[string]any) map[string]any {
188+ registrations := make([]any, 0, len(routes))
189+ for i, route := range routes {
190+ registrations = append(registrations, map[string]any{
191+ "endpoint": testEndpoint + "-" + string(rune('a'+i)),
192+ "incarnation": "0102030405060708090a0b0c0d0e0f10",
193+ "generation": 1,
194+ "identity": "offline-daemon-" + string(rune('a'+i)),
195+ "route": route,
196+ "policy": map[string]any{
197+ "protocolVersion": 1,
198+ "catalogSchemaVersion": 3,
199+ "environmentPolicy": "client-owned",
200+ "transport": "offline",
201+ "trust": "offline-trust",
202+ "launch": "offline-launch",
203+ "isolation": "offline-isolation",
204+ },
205+ })
206+ }
207+ return map[string]any{"marker": Marker, "registrations": registrations}
208+}
A internal/adapters/brokeripc/bounded_setup_test.go +328 -0
@@ -0,0 +1,328 @@
1+package brokeripc
2+
3+import (
4+ "context"
5+ "errors"
6+ "runtime"
7+ "sync"
8+ "testing"
9+ "time"
10+
11+ "github.com/stretchr/testify/require"
12+
13+ "github.com/bnema/vev/internal/adapters/brokerwire"
14+ "github.com/bnema/vev/internal/adapters/ipc"
15+)
16+
17+// Fully bounded broker setup (P3.4 slice A). These tests drive the real AF_UNIX
18+// carriage, never an in-memory replacement, so the only way to interrupt a peer
19+// parked in framing I/O is the behavior under test: the client closing the
20+// carriage when its single setup deadline expires, and the server enforcing the
21+// accept-time registration budget on the pre-Register phase.
22+//
23+// The client half: one Dial context/deadline bounds the Unix dial, the preamble
24+// exchange, the Register send, and the wait for Registered; a peer that stalls
25+// after a successful preamble is interrupted by closing the carriage with every
26+// setup worker joined; and a successful connection detaches from the setup
27+// context after Registered. The server half: after ports.BrokerAuthority
28+// admission a client must send Register within the same accept-time handshake
29+// budget, or the session, its admitted core lease, and its listener slot are
30+// released, while Listener.Close stays immediate.
31+
32+// silentBrokerPeer is one real AF_UNIX peer that completes the broker preamble
33+// and then never answers Register, so a client's registration wait is observable
34+// and interruptible without sleeping on a race.
35+type silentBrokerPeer struct {
36+ mux ipc.MuxListener
37+ preambleSeen chan struct{}
38+ release chan struct{}
39+ done chan struct{}
40+ once sync.Once
41+}
42+
43+// startSilentBrokerPeer binds one real same-user endpoint whose only peer
44+// completes the preamble and then stays silent.
45+func startSilentBrokerPeer(t *testing.T, path string) *silentBrokerPeer {
46+ t.Helper()
47+ mux, err := ipc.ListenMux(path)
48+ require.NoError(t, err)
49+ peer := &silentBrokerPeer{
50+ mux: mux,
51+ preambleSeen: make(chan struct{}),
52+ release: make(chan struct{}),
53+ done: make(chan struct{}),
54+ }
55+ go func() {
56+ defer close(peer.done)
57+ conn, err := mux.Accept()
58+ if err != nil {
59+ return
60+ }
61+ defer func() { _ = conn.Close() }()
62+ if _, err := runServerPreamble(context.Background(), conn, brokerwire.DefaultCeilings()); err != nil {
63+ return
64+ }
65+ close(peer.preambleSeen)
66+ <-peer.release
67+ }()
68+ t.Cleanup(peer.stop)
69+ return peer
70+}
71+
72+// stop releases the silent peer and joins it. It is idempotent.
73+func (p *silentBrokerPeer) stop() {
74+ p.once.Do(func() { close(p.release) })
75+ _ = p.mux.Close()
76+ <-p.done
77+}
78+
79+// awaitPreamble fails unless the peer observed the client's preamble.
80+func (p *silentBrokerPeer) awaitPreamble(t *testing.T) {
81+ t.Helper()
82+ select {
83+ case <-p.preambleSeen:
84+ case <-time.After(5 * time.Second):
85+ t.Fatal("the peer never observed the broker preamble")
86+ }
87+}
88+
89+// TestClientDialSilentAfterPreambleIsBoundedAndJoined proves a peer that
90+// completes the preamble and never answers Register cannot park Dial past the
91+// configured handshake budget: Dial returns a deadline failure, the carriage was
92+// closed to interrupt the blocked wait, and the setup workers were joined
93+// instead of being abandoned.
94+func TestClientDialSilentAfterPreambleIsBoundedAndJoined(t *testing.T) {
95+ path := testSocketPath(t)
96+ peer := startSilentBrokerPeer(t, path)
97+
98+ baseline := runtime.NumGoroutine()
99+ ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
100+ defer cancel()
101+ started := time.Now()
102+ client, err := Dial(ctx, path, Config{HandshakeTimeout: 200 * time.Millisecond})
103+ elapsed := time.Since(started)
104+ require.ErrorIs(t, err, context.DeadlineExceeded)
105+ require.Nil(t, client)
106+ require.Less(t, elapsed, 5*time.Second, "the setup deadline must interrupt the silent registration wait")
107+ peer.awaitPreamble(t)
108+
109+ peer.stop()
110+ require.Eventually(t, func() bool { return runtime.NumGoroutine() <= baseline+2 }, 5*time.Second, 5*time.Millisecond,
111+ "a bounded Dial must not abandon a setup goroutine (baseline=%d now=%d)", baseline, runtime.NumGoroutine())
112+}
113+
114+// TestClientDialEverySetupStepSharesTheOneDeadline proves the caller's context
115+// deadline, not only HandshakeTimeout, bounds the whole client setup: a deadline
116+// far tighter than HandshakeTimeout still interrupts the registration wait.
117+func TestClientDialEverySetupStepSharesTheOneDeadline(t *testing.T) {
118+ path := testSocketPath(t)
119+ peer := startSilentBrokerPeer(t, path)
120+
121+ ctx, cancel := context.WithTimeout(context.Background(), 150*time.Millisecond)
122+ defer cancel()
123+ started := time.Now()
124+ client, err := Dial(ctx, path, Config{HandshakeTimeout: 30 * time.Second})
125+ elapsed := time.Since(started)
126+ require.ErrorIs(t, err, context.DeadlineExceeded)
127+ require.Nil(t, client)
128+ require.Less(t, elapsed, 5*time.Second, "the caller's deadline must bound the setup")
129+ peer.awaitPreamble(t)
130+ peer.stop()
131+}
132+
133+// TestClientDialCancellationInterruptsRegistrationWait proves an explicit
134+// cancellation, not just a deadline, interrupts the wait for Registered and is
135+// reported as cancellation rather than being masked by a carriage failure.
136+func TestClientDialCancellationInterruptsRegistrationWait(t *testing.T) {
137+ path := testSocketPath(t)
138+ peer := startSilentBrokerPeer(t, path)
139+
140+ ctx, cancel := context.WithCancel(context.Background())
141+ defer cancel()
142+ result := make(chan error, 1)
143+ go func() {
144+ _, err := Dial(ctx, path, Config{HandshakeTimeout: 30 * time.Second})
145+ result <- err
146+ }()
147+ peer.awaitPreamble(t)
148+ cancel()
149+
150+ select {
151+ case err := <-result:
152+ require.ErrorIs(t, err, context.Canceled, "cancellation must interrupt the registration wait")
153+ case <-time.After(5 * time.Second):
154+ t.Fatal("Dial must return on cancellation instead of waiting out HandshakeTimeout")
155+ }
156+ peer.stop()
157+}
158+
159+// TestClientConnectionOutlivesSetupContext proves a successful connection
160+// detaches from the setup context after Registered: once Dial returns, the
161+// caller's setup deadline may elapse without disturbing the established
162+// connection.
163+func TestClientConnectionOutlivesSetupContext(t *testing.T) {
164+ e := startEndpoint(t, Config{HandshakeTimeout: 5 * time.Second})
165+
166+ setupCtx, cancelSetup := context.WithTimeout(context.Background(), 200*time.Millisecond)
167+ defer cancelSetup()
168+ client, err := Dial(setupCtx, e.path, Config{HandshakeTimeout: 5 * time.Second})
169+ require.NoError(t, err)
170+ t.Cleanup(func() { _ = client.Close() })
171+ session := e.accept()
172+ require.Equal(t, session.ConnectionID(), client.ConnectionID())
173+
174+ // Let the setup deadline elapse; the connection was registered in time and
175+ // must keep serving on its own connection-lived context.
176+ time.Sleep(400 * time.Millisecond)
177+ require.ErrorIs(t, setupCtx.Err(), context.DeadlineExceeded, "the setup context must have expired")
178+
179+ ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
180+ defer cancel()
181+ require.NoError(t, client.AddHost(ctx, "after@setup:22"), "a registered connection must outlive its setup context")
182+}
183+
184+// TestServerRegistrationBudgetSettlesSilentPeer proves a same-user peer cannot
185+// hold an admitted core lease or a listener slot indefinitely: after
186+// ports.BrokerAuthority admission the client must send Register within the
187+// accept-time handshake budget, or the session closes the core service and
188+// releases the slot, and the listener keeps serving.
189+func TestServerRegistrationBudgetSettlesSilentPeer(t *testing.T) {
190+ e := startEndpoint(t, Config{MaxClients: 1, HandshakeTimeout: 200 * time.Millisecond})
191+ l := e.listener.(*listener)
192+
193+ ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
194+ defer cancel()
195+ transport, err := ipc.DialMuxContext(ctx, e.path)
196+ require.NoError(t, err)
197+ t.Cleanup(func() { _ = transport.Close() })
198+ _, err = runClientPreamble(ctx, transport, brokerwire.DefaultCeilings())
199+ require.NoError(t, err)
200+
201+ session := e.accept()
202+ core := e.authority.last()
203+ require.NotNil(t, core)
204+
205+ require.Eventually(t, func() bool {
206+ select {
207+ case <-core.done:
208+ return true
209+ default:
210+ return false
211+ }
212+ }, 5*time.Second, 5*time.Millisecond, "a silent unregistered peer must not hold the admitted core lease")
213+ require.Eventually(t, func() bool { return l.liveSessions() == 0 }, 5*time.Second, 5*time.Millisecond,
214+ "a settled session must leave the listener's live set")
215+
216+ // The timeout is the peer's departure, not a session failure: the settled
217+ // session reports an orderly end even though its terminal cause still records
218+ // the registration budget.
219+ require.NoError(t, session.Close())
220+ require.ErrorIs(t, session.(*serverSession).terminalErr(), ErrRegistrationTimeout)
221+ require.ErrorIs(t, session.(*serverSession).terminalErr(), context.DeadlineExceeded)
222+
223+ // The released slot is reusable even at the bound of one: the listener
224+ // admits a fresh same-user client after the timeout.
225+ fresh := e.dial()
226+ freshSession := e.accept()
227+ require.Equal(t, fresh.ConnectionID(), freshSession.ConnectionID())
228+
229+ // A clean listener shutdown must not report a peer timeout as its own
230+ // failure, for the settled session or for the live one it ends.
231+ require.NoError(t, e.listener.Close())
232+}
233+
234+// TestRegistrationDeadlineBoundaryMarker proves the pre-Register guard treats
235+// the registered marker, not timer ordering, as the deadline boundary: a session
236+// whose Register handling already began is never settled with
237+// ErrRegistrationTimeout, while a session with no Register is. The deadline is
238+// already in the past so the guard observes the boundary deterministically.
239+func TestRegistrationDeadlineBoundaryMarker(t *testing.T) {
240+ tests := []struct {
241+ name string
242+ registered bool
243+ wantTimeout bool
244+ }{
245+ {name: "register handling begun at the deadline", registered: true},
246+ {name: "no register by the deadline", registered: false, wantTimeout: true},
247+ }
248+ for _, tc := range tests {
249+ t.Run(tc.name, func(t *testing.T) {
250+ core := newTestCore(0x51)
251+ session, err := newServerSession(0x51, newRecordingTransport(), brokerwire.DefaultCeilings(), core, Config{}, func() {}, time.Now().Add(-time.Second))
252+ require.NoError(t, err)
253+ session.registered.Store(tc.registered)
254+
255+ stop := session.armRegistrationDeadline()
256+ if tc.wantTimeout {
257+ require.Eventually(t, func() bool { return errors.Is(session.terminalErr(), ErrRegistrationTimeout) }, time.Second, time.Millisecond)
258+ }
259+ stop()
260+
261+ if tc.wantTimeout {
262+ require.ErrorIs(t, session.terminalErr(), ErrRegistrationTimeout)
263+ require.ErrorIs(t, session.terminalErr(), context.DeadlineExceeded)
264+ return
265+ }
266+ require.NoError(t, session.terminalErr(), "a Register already being handled must not be settled as a peer timeout")
267+ })
268+ }
269+}
270+
271+// TestServerRegistrationBudgetLeavesRegisteredSessionHealthy proves the
272+// registration budget bounds only the pre-Register phase: a session that
273+// registered in time keeps serving long after the accept-time deadline has
274+// passed.
275+func TestServerRegistrationBudgetLeavesRegisteredSessionHealthy(t *testing.T) {
276+ e := startEndpoint(t, Config{HandshakeTimeout: 150 * time.Millisecond})
277+ client, _ := e.pair()
278+ core := e.authority.last()
279+
280+ // Outlive the accept-time registration budget.
281+ time.Sleep(400 * time.Millisecond)
282+ ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
283+ defer cancel()
284+ require.NoError(t, client.AddHost(ctx, "after@budget:22"))
285+ select {
286+ case <-core.done:
287+ t.Fatal("a session that registered in time must not be settled by the registration budget")
288+ default:
289+ }
290+}
291+
292+// TestListenerCloseDuringRegistrationWindowIsImmediate proves Close remains
293+// immediate for a session that is inside its registration window with a
294+// deliberately long budget: it does not wait out HandshakeTimeout, and it
295+// releases the admitted core service and the slot.
296+func TestListenerCloseDuringRegistrationWindowIsImmediate(t *testing.T) {
297+ e := startEndpoint(t, Config{MaxClients: 2, HandshakeTimeout: 30 * time.Second})
298+ l := e.listener.(*listener)
299+
300+ ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
301+ defer cancel()
302+ transport, err := ipc.DialMuxContext(ctx, e.path)
303+ require.NoError(t, err)
304+ t.Cleanup(func() { _ = transport.Close() })
305+ _, err = runClientPreamble(ctx, transport, brokerwire.DefaultCeilings())
306+ require.NoError(t, err)
307+
308+ _ = e.accept()
309+ core := e.authority.last()
310+ require.NotNil(t, core)
311+
312+ closed := make(chan error, 1)
313+ go func() { closed <- e.listener.Close() }()
314+ select {
315+ case err := <-closed:
316+ require.NoError(t, err)
317+ case <-time.After(5 * time.Second):
318+ t.Fatal("Close must not wait out the registration budget for a silent session")
319+ }
320+ select {
321+ case <-core.done:
322+ case <-time.After(5 * time.Second):
323+ t.Fatal("Close must end the admitted core service")
324+ }
325+ require.Zero(t, l.liveSessions())
326+ require.Eventually(t, func() bool { return len(l.slots) == 0 }, 5*time.Second, 5*time.Millisecond,
327+ "Close must release the client slot")
328+}
M internal/adapters/brokeripc/client.go +46 -8
@@ -45,22 +45,31 @@ func dial(ctx context.Context, path string, cfg Config, verify ipc.PeerVerifier)
45 if path == "" {45 if path == "" {
46 return nil, ErrConfig46 return nil, ErrConfig
47 }47 }
48- transport, err := ipc.DialMuxContextWithPeerVerifier(ctx, path, verify)48+ // One deadline bounds the whole setup: the Unix dial, the preamble
49+ // exchange, the Register send, and the wait for Registered. Each setup step
50+ // closes the carriage when that deadline expires, because a context alone
51+ // cannot interrupt blocking framing I/O; that is what stops a peer that
52+ // stalls after a successful preamble from parking Dial forever.
53+ setup, cancelSetup := context.WithTimeout(ctx, cfg.HandshakeTimeout)
54+ defer cancelSetup()
55+ transport, err := ipc.DialMuxContextWithPeerVerifier(setup, path, verify)
49 if err != nil {56 if err != nil {
50 return nil, err57 return nil, err
51 }58 }
52- handshake, cancel := context.WithTimeout(ctx, cfg.HandshakeTimeout)59+ ceilings, err := runClientPreamble(setup, transport, brokerwire.DefaultCeilings())
53- defer cancel()
54- ceilings, err := runClientPreamble(handshake, transport, brokerwire.DefaultCeilings())
55 if err != nil {60 if err != nil {
56 _ = transport.Close()61 _ = transport.Close()
57 return nil, err62 return nil, err
58 }63 }
59- scope, err := register(handshake, transport, ceilings)64+ scope, err := register(setup, transport, ceilings)
60 if err != nil {65 if err != nil {
61 _ = transport.Close()66 _ = transport.Close()
62 return nil, err67 return nil, err
63 }68 }
69+ // Setup is complete and the carriage is healthy, so the established
70+ // connection detaches from the setup deadline: it gets its own
71+ // connection-lived context below, and cancelSetup (deferred) only ends an
72+ // already-finished setup.
64 conn, err := brokerwire.NewConnection(scope)73 conn, err := brokerwire.NewConnection(scope)
65 if err != nil {74 if err != nil {
66 _ = transport.Close()75 _ = transport.Close()
@@ -94,7 +103,39 @@ func dial(ctx context.Context, path string, cfg Config, verify ipc.PeerVerifier)
94 // register completes the broker registration exchange: one Register follows the103 // register completes the broker registration exchange: one Register follows the
95 // preamble, and the first server frame must be the Registered answer carrying104 // preamble, and the first server frame must be the Registered answer carrying
96 // the scope assigned at accept. Any other first frame is a protocol violation.105 // the scope assigned at accept. Any other first frame is a protocol violation.
106+//
107+// The wait is bounded by ctx: a peer that stalls after a successful preamble is
108+// interrupted by closing the carriage, and the exchange worker is joined before
109+// returning, so no setup goroutine is abandoned on the carriage. A Registered
110+// observed after ctx expired is not a success.
97 func register(ctx context.Context, transport wire.BoundedTransport, ceilings brokerwire.Ceilings) (brokerwire.Scope, error) {111 func register(ctx context.Context, transport wire.BoundedTransport, ceilings brokerwire.Ceilings) (brokerwire.Scope, error) {
112+ type result struct {
113+ scope brokerwire.Scope
114+ err error
115+ }
116+ done := make(chan result, 1)
117+ go func() {
118+ scope, err := exchangeRegister(transport, ceilings)
119+ done <- result{scope: scope, err: err}
120+ }()
121+ select {
122+ case <-ctx.Done():
123+ _ = transport.Close()
124+ <-done
125+ return brokerwire.Scope{}, ctx.Err()
126+ case outcome := <-done:
127+ if outcome.err == nil {
128+ if err := ctx.Err(); err != nil {
129+ return brokerwire.Scope{}, err
130+ }
131+ }
132+ return outcome.scope, outcome.err
133+ }
134+}
135+
136+// exchangeRegister runs one Register/Registered round trip on the carriage. It
137+// is synchronous: the caller bounds it by closing the carriage on ctx expiry.
138+func exchangeRegister(transport wire.BoundedTransport, ceilings brokerwire.Ceilings) (brokerwire.Scope, error) {
98 payload, err := brokerwire.EncodeClient(brokerwire.Register{}, ceilings.MaxReceiveEnvelopeBytes, ceilings.StreamChunkLimit)139 payload, err := brokerwire.EncodeClient(brokerwire.Register{}, ceilings.MaxReceiveEnvelopeBytes, ceilings.StreamChunkLimit)
99 if err != nil {140 if err != nil {
100 return brokerwire.Scope{}, err141 return brokerwire.Scope{}, err
@@ -118,9 +159,6 @@ func register(ctx context.Context, transport wire.BoundedTransport, ceilings bro
118 if err := scope.Validate(); err != nil {159 if err := scope.Validate(); err != nil {
119 return brokerwire.Scope{}, errors.Join(ErrProtocol, err)160 return brokerwire.Scope{}, errors.Join(ErrProtocol, err)
120 }161 }
121- if err := ctx.Err(); err != nil {
122- return brokerwire.Scope{}, err
123- }
124 return scope, nil162 return scope, nil
125 }163 }
126 164
M internal/adapters/brokeripc/config.go +9 -4
@@ -33,10 +33,15 @@ type Config struct {
33 StreamInboundChunks int33 StreamInboundChunks int
34 // StreamInboundBytes bounds one logical stream's inbound chunk bytes.34 // StreamInboundBytes bounds one logical stream's inbound chunk bytes.
35 StreamInboundBytes uint6435 StreamInboundBytes uint64
36- // HandshakeTimeout bounds one accepted connection's broker preamble plus36+ // HandshakeTimeout bounds one accepted connection's broker preamble, broker
37- // admission. The client dialer also bounds its own preamble exchange with37+ // admission, and the wait for its Register. The client dialer also bounds
38- // it. It is deliberately separate from the session handshake budget that38+ // its whole setup (Unix dial, preamble, Register send, and wait for
39- // runs inside a logical stream.39+ // Registered) with it. On the server the registration phase shares the same
40+ // accept-time budget as the preamble and admission: a peer that completes
41+ // the preamble and admission but never sends Register is settled when that
42+ // deadline elapses, so it cannot hold a core client lease or a listener slot
43+ // indefinitely. It is deliberately separate from the session handshake
44+ // budget that runs inside a logical stream.
40 HandshakeTimeout time.Duration45 HandshakeTimeout time.Duration
41 // MaxPendingOperations bounds mutating operations this side is waiting for46 // MaxPendingOperations bounds mutating operations this side is waiting for
42 // on one connection. It never exceeds the brokerwire per-connection bound.47 // on one connection. It never exceeds the brokerwire per-connection bound.
M internal/adapters/brokeripc/doc.go +17 -5
@@ -24,7 +24,17 @@
24 // ceilings, and waits for exactly one Register before admitting subscription,24 // ceilings, and waits for exactly one Register before admitting subscription,
25 // operation, or stream work; every later frame must carry the assigned epoch25 // operation, or stream work; every later frame must carry the assigned epoch
26 // and connection identity or it is refused as stale without touching session26 // and connection identity or it is refused as stale without touching session
27-// state. Bounds are explicit: accepted clients are capped per listener, stream27+// state. Setup is fully bounded on both halves. The client dialer bounds the
28+// Unix dial, the preamble exchange, the Register send, and the wait for
29+// Registered with one context/deadline; a context cannot interrupt blocking
30+// framing I/O, so a setup step closes the carriage on expiry and joins its
31+// worker, which is what stops a peer that stalls after a successful preamble,
32+// and a connection that registered detaches from the setup context. Symmetrically
33+// the server requires the client's Register within the same accept-time
34+// handshake budget that bounded the preamble and admission, so a same-user peer
35+// cannot complete the preamble, take an admitted core lease, and then hold that
36+// lease and its listener slot indefinitely by staying silent. Bounds are
37+// explicit: accepted clients are capped per listener, stream
28 // inbound queues are bounded per stream, pending operations are capped per38 // inbound queues are bounded per stream, pending operations are capped per
29 // connection, and the brokerwire per-connection trackers bound streams and39 // connection, and the brokerwire per-connection trackers bound streams and
30 // completed-operation dedup. Every accepted connection owns its own reader,40 // completed-operation dedup. Every accepted connection owns its own reader,
@@ -39,8 +49,10 @@
39 // Close returns.49 // Close returns.
40 //50 //
41 // Like the brokerwire codec (P3.1) and the daemonmux carriage (P3.2), this51 // Like the brokerwire codec (P3.1) and the daemonmux carriage (P3.2), this
42-// adapter is not activated by production composition. Nothing in internal/app52+// adapter is not activated by production composition: no ordinary command,
43-// or main constructs it yet; P3.4 composes the broker use case, its stores, and53+// path, or factory constructs it. P3.4 composes the broker use case, its
44-// this listener behind a hidden entry point, and P7 performs the coordinated54+// stores, and this listener behind the hidden `_broker-serve` sandbox entry
45-// production cutover.55+// point, and slice D adds the hidden `_broker-launcher` and `_broker-status`
56+// detached connect-or-spawn helpers over the same private root; P7 performs the
57+// coordinated production cutover.
46 package brokeripc58 package brokeripc
M internal/adapters/brokeripc/errors.go +7 -0
@@ -44,6 +44,13 @@ var (
44 // that is not the one assigned at accept. It is refused as stale and never44 // that is not the one assigned at accept. It is refused as stale and never
45 // applied.45 // applied.
46 ErrScopeMismatch = errors.New("brokeripc: frame scope does not match the accepted connection")46 ErrScopeMismatch = errors.New("brokeripc: frame scope does not match the accepted connection")
47+ // ErrRegistrationTimeout reports a client that completed the broker preamble
48+ // and admission but did not send its Register within the accept-time
49+ // handshake budget. The session is settled, its admitted core service is
50+ // closed, and its listener slot is released; context.DeadlineExceeded also
51+ // matches it. It is an orderly peer disconnect, so a listener draining its
52+ // sessions does not report a clean shutdown as this timeout.
53+ ErrRegistrationTimeout = errors.New("brokeripc: broker registration timed out")
47 )54 )
48 55
49 // transportFailure classifies one carriage error that ended a connection. A56 // transportFailure classifies one carriage error that ended a connection. A
M internal/adapters/brokeripc/limits_test.go +12 -4
@@ -4,7 +4,10 @@ import (
4 "context"4 "context"
5 "errors"5 "errors"
6 "io"6 "io"
7+ "io/fs"
8+ "net"
7 "sync"9 "sync"
10+ "syscall"
8 "testing"11 "testing"
9 "time"12 "time"
10 13
@@ -273,21 +276,26 @@ func (unboundedTransport) Close() error { return nil }
273 // never serve.276 // never serve.
274 func TestServerSessionRejectsInvalidConstruction(t *testing.T) {277 func TestServerSessionRejectsInvalidConstruction(t *testing.T) {
275 core := &fakeCore{id: ports.BrokerConnectionID{1}, hub: newSnapshotHub(ports.BrokerSnapshot{}), streams: map[ports.BrokerStreamID]*fakeLogicalConn{}, done: make(chan struct{})}278 core := &fakeCore{id: ports.BrokerConnectionID{1}, hub: newSnapshotHub(ports.BrokerSnapshot{}), streams: map[ports.BrokerStreamID]*fakeLogicalConn{}, done: make(chan struct{})}
276- _, err := newServerSession(0, nil, brokerwire.DefaultCeilings(), core, Config{}, nil)279+ _, err := newServerSession(0, nil, brokerwire.DefaultCeilings(), core, Config{}, nil, time.Time{})
277 require.ErrorIs(t, err, ErrConfig)280 require.ErrorIs(t, err, ErrConfig)
278- _, err = newServerSession(1, nil, brokerwire.DefaultCeilings(), nil, Config{}, nil)281+ _, err = newServerSession(1, nil, brokerwire.DefaultCeilings(), nil, Config{}, nil, time.Time{})
279 require.ErrorIs(t, err, ErrConfig)282 require.ErrorIs(t, err, ErrConfig)
280 require.NoError(t, brokerwire.DefaultCeilings().Validate())283 require.NoError(t, brokerwire.DefaultCeilings().Validate())
281 require.Error(t, brokerwire.Ceilings{}.Validate())284 require.Error(t, brokerwire.Ceilings{}.Validate())
282 }285 }
283 286
284-// TestOrderlyDisconnectClassification proves a peer disconnect and a local close287+// TestOrderlyDisconnectClassification proves a peer disconnect, a local close,
285-// are orderly ends while a protocol violation is not.288+// and a peer registration timeout are orderly ends while a protocol violation
289+// is not.
286 func TestOrderlyDisconnectClassification(t *testing.T) {290 func TestOrderlyDisconnectClassification(t *testing.T) {
287 require.True(t, orderlyDisconnect(nil))291 require.True(t, orderlyDisconnect(nil))
288 require.True(t, orderlyDisconnect(io.EOF))292 require.True(t, orderlyDisconnect(io.EOF))
289 require.True(t, orderlyDisconnect(ErrSessionClosed))293 require.True(t, orderlyDisconnect(ErrSessionClosed))
290 require.True(t, orderlyDisconnect(ErrConnectionClosed))294 require.True(t, orderlyDisconnect(ErrConnectionClosed))
295+ require.True(t, orderlyDisconnect(errors.Join(ErrRegistrationTimeout, context.DeadlineExceeded)))
296+ require.True(t, orderlyDisconnect(&net.OpError{Op: "read", Net: "unix", Err: syscall.ECONNRESET}))
297+ require.True(t, orderlyDisconnect(net.ErrClosed))
298+ require.True(t, orderlyDisconnect(fs.ErrClosed))
291 require.False(t, orderlyDisconnect(ErrMalformedFrame))299 require.False(t, orderlyDisconnect(ErrMalformedFrame))
292 require.False(t, orderlyDisconnect(ErrProtocol))300 require.False(t, orderlyDisconnect(ErrProtocol))
293 }301 }
M internal/adapters/brokeripc/listen_test.go +1 -1
@@ -21,7 +21,7 @@ import (
21 // gatedAuthority blocks admission until a test releases it, so a listener Close21 // gatedAuthority blocks admission until a test releases it, so a listener Close
22 // can race an admission that has already reached the authority.22 // can race an admission that has already reached the authority.
23 type gatedAuthority struct {23 type gatedAuthority struct {
24- inner Authority24+ inner ports.BrokerAuthority
25 entered chan struct{}25 entered chan struct{}
26 gate chan struct{}26 gate chan struct{}
27 once sync.Once27 once sync.Once
M internal/adapters/brokeripc/preamble.go +12 -2
@@ -21,11 +21,16 @@ import (
21 //21 //
22 // The preamble envelope is bounded to brokerwire.BrokerPreambleLimit (4 KiB)22 // The preamble envelope is bounded to brokerwire.BrokerPreambleLimit (4 KiB)
23 // before allocation, strict-scanned with wire.ScanEnvelope before generated23 // before allocation, strict-scanned with wire.ScanEnvelope before generated
24-// unmarshal, and is never restarted: the caller's context (the configured24+// unmarshal, and is never restarted. The caller's context (the configured
25-// handshake timeout) bounds the whole exchange.25+// setup/handshake budget) bounds the whole exchange: a context alone cannot
26+// interrupt the blocking carriage I/O, so the client half closes the carriage
27+// on expiry to release the worker and joins it before returning.
26 28
27 // runClientPreamble sends the local broker PreambleRequest offer and accepts29 // runClientPreamble sends the local broker PreambleRequest offer and accepts
28 // exactly one PreambleResponse. It returns the effective negotiated ceilings.30 // exactly one PreambleResponse. It returns the effective negotiated ceilings.
31+// On ctx expiry it closes the carriage, which interrupts the blocking send or
32+// receive, and joins its worker before returning, so a peer that stalls
33+// mid-preamble never leaves a setup goroutine parked on the carriage.
29 func runClientPreamble(ctx context.Context, transport wire.Transport, offer brokerwire.Ceilings) (brokerwire.Ceilings, error) {34 func runClientPreamble(ctx context.Context, transport wire.Transport, offer brokerwire.Ceilings) (brokerwire.Ceilings, error) {
30 if err := offer.Validate(); err != nil {35 if err := offer.Validate(); err != nil {
31 return brokerwire.Ceilings{}, errors.Join(ErrConfig, err)36 return brokerwire.Ceilings{}, errors.Join(ErrConfig, err)
@@ -69,6 +74,11 @@ func runClientPreamble(ctx context.Context, transport wire.Transport, offer brok
69 }()74 }()
70 select {75 select {
71 case <-ctx.Done():76 case <-ctx.Done():
77+ // A context alone cannot reach a Send or RecvBounded parked on a silent
78+ // peer; closing the carriage releases the worker, and draining done
79+ // joins it before returning. No setup goroutine outlives the exchange.
80+ _ = transport.Close()
81+ <-done
72 return brokerwire.Ceilings{}, ctx.Err()82 return brokerwire.Ceilings{}, ctx.Err()
73 case outcome := <-done:83 case outcome := <-done:
74 return outcome.ceilings, outcome.err84 return outcome.ceilings, outcome.err
M internal/adapters/brokeripc/server.go +150 -40
@@ -5,7 +5,12 @@ import (
5 "errors"5 "errors"
6 "fmt"6 "fmt"
7 "io"7 "io"
8+ "io/fs"
9+ "net"
8 "sync"10 "sync"
11+ "sync/atomic"
12+ "syscall"
13+ "time"
9 14
10 "github.com/bnema/vev/internal/adapters/brokerwire"15 "github.com/bnema/vev/internal/adapters/brokerwire"
11 "github.com/bnema/vev/internal/adapters/ipc"16 "github.com/bnema/vev/internal/adapters/ipc"
@@ -30,24 +35,6 @@ import (
30 // one goroutine per admitted mutating operation. Every one of those is35 // one goroutine per admitted mutating operation. Every one of those is
31 // per-connection, so a slow or stalled client blocks only its own work.36 // per-connection, so a slow or stalled client blocks only its own work.
32 37
33-// Authority admits one accepted broker client to the broker core. The returned
34-// ports.BrokerService is the core's service for exactly that connection: its
35-// ConnectionID is the identity the session stamps on every frame, and Close
36-// releases the connection's core resources. P3.4 supplies the broker use case
37-// here; this adapter never constructs a broker itself.
38-//
39-// ctx bounds AdmitClient only. It carries the listener's handshake deadline and
40-// is canceled either as soon as admission returns or when the listener closes,
41-// so an implementation must not retain it or start work that outlives admission
42-// from it. Honoring ctx cancellation is what lets Listener.Close drop a client
43-// already parked in admission immediately instead of waiting out the handshake
44-// budget. The admitted service gets its own connection-lived context from the
45-// session. An authority that needs a longer-lived context must supply its own
46-// rather than keep this one.
47-type Authority interface {
48- AdmitClient(ctx context.Context) (ports.BrokerService, error)
49-}
50-
51 // Listen binds the private per-user broker endpoint at path and returns a38 // Listen binds the private per-user broker endpoint at path and returns a
52 // ports.BrokerListener that admits same-user clients only.39 // ports.BrokerListener that admits same-user clients only.
53 //40 //
@@ -57,14 +44,22 @@ type Authority interface {
57 // without being removed, a stale socket left by a dead owner is recovered44 // without being removed, a stale socket left by a dead owner is recovered
58 // race-safely, and a live owner is reported as ipc.ErrDaemonRunning instead of45 // race-safely, and a live owner is reported as ipc.ErrDaemonRunning instead of
59 // being evicted.46 // being evicted.
60-func Listen(path string, epoch ports.BrokerEpoch, authority Authority, cfg Config, opts ...ipc.Option) (ports.BrokerListener, error) {47+//
48+// authority is the ports.BrokerAuthority admission seam: the listener calls
49+// AdmitClient once per accepted connection under the handshake budget and binds
50+// the returned service to exactly that connection. That service's ConnectionID
51+// is the identity the session stamps on every frame; the P3.4 broker use case
52+// supplies the implementation, and this adapter consumes it without
53+// constructing a broker itself. See ports.BrokerAuthority for the
54+// admission-context contract.
55+func Listen(path string, epoch ports.BrokerEpoch, authority ports.BrokerAuthority, cfg Config, opts ...ipc.Option) (ports.BrokerListener, error) {
61 return listen(path, epoch, authority, cfg, ipc.SameUserPeerVerifier(), opts...)56 return listen(path, epoch, authority, cfg, ipc.SameUserPeerVerifier(), opts...)
62 }57 }
63 58
64 // listen is Listen with an injected peer verifier, so a test can observe a59 // listen is Listen with an injected peer verifier, so a test can observe a
65 // deterministic same-user refusal without root. A nil verifier falls back to60 // deterministic same-user refusal without root. A nil verifier falls back to
66 // the platform default.61 // the platform default.
67-func listen(path string, epoch ports.BrokerEpoch, authority Authority, cfg Config, verify ipc.PeerVerifier, opts ...ipc.Option) (ports.BrokerListener, error) {62+func listen(path string, epoch ports.BrokerEpoch, authority ports.BrokerAuthority, cfg Config, verify ipc.PeerVerifier, opts ...ipc.Option) (ports.BrokerListener, error) {
68 cfg = cfg.withDefaults()63 cfg = cfg.withDefaults()
69 if err := cfg.validate(); err != nil {64 if err := cfg.validate(); err != nil {
70 return nil, err65 return nil, err
@@ -110,7 +105,7 @@ func listen(path string, epoch ports.BrokerEpoch, authority Authority, cfg Confi
110 // already-accepted client immediately instead of waiting out HandshakeTimeout.105 // already-accepted client immediately instead of waiting out HandshakeTimeout.
111 type listener struct {106 type listener struct {
112 epoch ports.BrokerEpoch107 epoch ports.BrokerEpoch
113- authority Authority108+ authority ports.BrokerAuthority
114 cfg Config109 cfg Config
115 mux ipc.MuxListener110 mux ipc.MuxListener
116 slots chan struct{}111 slots chan struct{}
@@ -338,9 +333,9 @@ func (l *listener) Accept() (ports.BrokerService, error) {
338 // startSession completes the preamble and admission for one accepted carriage333 // startSession completes the preamble and admission for one accepted carriage
339 // and starts the session. The admission context is bounded to the handshake334 // and starts the session. The admission context is bounded to the handshake
340 // budget and is canceled as soon as admission returns or when the listener335 // budget and is canceled as soon as admission returns or when the listener
341-// closes, so the admitted service must not retain it (see Authority) and336+// closes, so the admitted service must not retain it (see
342-// Close can drop a client still in preamble or admission. Slot ownership337+// ports.BrokerAuthority) and Close can drop a client still in preamble or
343-// transfers to the session on success.338+// admission. Slot ownership transfers to the session on success.
344 func (l *listener) startSession(transport wire.BoundedTransport, release func()) (*serverSession, error) {339 func (l *listener) startSession(transport wire.BoundedTransport, release func()) (*serverSession, error) {
345 // The handshake budget is bounded by the listener-scoped context, so340 // The handshake budget is bounded by the listener-scoped context, so
346 // Listener.Close cancels a preamble or admission that is still waiting341 // Listener.Close cancels a preamble or admission that is still waiting
@@ -361,7 +356,7 @@ func (l *listener) startSession(transport wire.BoundedTransport, release func())
361 _ = transport.Close()356 _ = transport.Close()
362 return nil, fmt.Errorf("%w: authority admitted a nil service", ErrConfig)357 return nil, fmt.Errorf("%w: authority admitted a nil service", ErrConfig)
363 }358 }
364- session, err := newServerSession(l.epoch, transport, ceilings, core, l.cfg, release)359+ session, err := newServerSession(l.epoch, transport, ceilings, core, l.cfg, release, registrationDeadline(ctx))
365 if err != nil {360 if err != nil {
366 _ = core.Close()361 _ = core.Close()
367 _ = transport.Close()362 _ = transport.Close()
@@ -496,6 +491,23 @@ type serverSession struct {
496 // session from its live set. It is set before run starts.491 // session from its live set. It is set before run starts.
497 onShutdown func()492 onShutdown func()
498 493
494+ // registerDeadline is the absolute instant by which the client's Register
495+ // must arrive: the accept-time handshake deadline that bounded the preamble
496+ // and admission. Until Register is accepted the session is guarded by it, so
497+ // a peer that stays silent after admission is settled instead of pinning the
498+ // admitted core lease and the listener slot. Once Register is accepted the
499+ // guard is stopped and the session runs on its own connection-lived context.
500+ registerDeadline time.Time
501+ // registered is set atomically the instant Register handling begins, before
502+ // any connection state is touched. The pre-Register guard reads it when the
503+ // registration deadline elapses, so a Register that arrives at the deadline
504+ // boundary is never settled as a peer timeout even if the guard wins the
505+ // race to the deadline.
506+ registered atomic.Bool
507+ // stopRegistration stops and joins the pre-Register guard. It is set by run
508+ // and called once from the Register dispatch; a no-op for a zero deadline.
509+ stopRegistration func()
510+
499 errMu sync.Mutex511 errMu sync.Mutex
500 closeErr error512 closeErr error
501 done chan struct{}513 done chan struct{}
@@ -503,8 +515,27 @@ type serverSession struct {
503 515
504 var _ ports.BrokerService = (*serverSession)(nil)516 var _ ports.BrokerService = (*serverSession)(nil)
505 517
518+// registrationDeadline returns the absolute instant by which an accepted client
519+// must send its Register: the accept-time handshake deadline that already
520+// bounded the preamble and admission. The session extends that same budget over
521+// the pre-Register wait, so a peer that completes the preamble and admission and
522+// then stays silent cannot hold a core client lease or a listener slot past the
523+// configured handshake budget. A missing deadline yields the zero time, which
524+// disables the bound.
525+func registrationDeadline(ctx context.Context) time.Time {
526+ deadline, ok := ctx.Deadline()
527+ if !ok {
528+ return time.Time{}
529+ }
530+ return deadline
531+}
532+
506 // newServerSession binds one admitted connection to its exact scope.533 // newServerSession binds one admitted connection to its exact scope.
507-func newServerSession(epoch ports.BrokerEpoch, transport wire.BoundedTransport, ceilings brokerwire.Ceilings, core ports.BrokerService, cfg Config, release func()) (*serverSession, error) {534+//
535+// registerDeadline is the absolute instant by which the client's Register must
536+// arrive (the accept-time handshake deadline); the zero time disables the bound
537+// and exists only for direct construction outside the listener.
538+func newServerSession(epoch ports.BrokerEpoch, transport wire.BoundedTransport, ceilings brokerwire.Ceilings, core ports.BrokerService, cfg Config, release func(), registerDeadline time.Time) (*serverSession, error) {
508 if epoch == 0 || transport == nil || core == nil {539 if epoch == 0 || transport == nil || core == nil {
509 return nil, ErrConfig540 return nil, ErrConfig
510 }541 }
@@ -530,18 +561,19 @@ func newServerSession(epoch ports.BrokerEpoch, transport wire.BoundedTransport,
530 release = func() {}561 release = func() {}
531 }562 }
532 return &serverSession{563 return &serverSession{
533- epoch: epoch,564+ epoch: epoch,
534- scope: conn.Scope(),565+ scope: conn.Scope(),
535- conn: conn,566+ conn: conn,
536- core: core,567+ core: core,
537- transport: transport,568+ transport: transport,
538- ceilings: ceilings,569+ ceilings: ceilings,
539- cfg: cfg,570+ cfg: cfg,
540- release: release,571+ release: release,
541- ctx: ctx,572+ ctx: ctx,
542- cancel: cancel,573+ cancel: cancel,
543- streams: make(map[ports.BrokerStreamID]*serverStream),574+ streams: make(map[ports.BrokerStreamID]*serverStream),
544- done: make(chan struct{}),575+ registerDeadline: registerDeadline,
576+ done: make(chan struct{}),
545 }, nil577 }, nil
546 }578 }
547 579
@@ -551,6 +583,13 @@ func newServerSession(epoch ports.BrokerEpoch, transport wire.BoundedTransport,
551 // carriage error settles exactly this connection.583 // carriage error settles exactly this connection.
552 func (s *serverSession) run() {584 func (s *serverSession) run() {
553 defer s.shutdown()585 defer s.shutdown()
586+ // The pre-Register phase shares the accept-time handshake deadline that
587+ // bounded the preamble and admission. Arm it before the read loop and stop
588+ // (and join) it as soon as Register is accepted, so a silent peer cannot pin
589+ // the admitted core lease or the listener slot, and a healthy session is
590+ // never disturbed by it.
591+ s.stopRegistration = s.armRegistrationDeadline()
592+ defer s.stopRegistration()
554 for {593 for {
555 envelope, err := s.transport.RecvBounded(s.ceilings.MaxReceiveEnvelopeBytes)594 envelope, err := s.transport.RecvBounded(s.ceilings.MaxReceiveEnvelopeBytes)
556 if err != nil {595 if err != nil {
@@ -569,15 +608,72 @@ func (s *serverSession) run() {
569 }608 }
570 }609 }
571 610
611+// armRegistrationDeadline enforces the accept-time registration budget on the
612+// pre-Register phase. The peer must send its Register before the absolute
613+// deadline that already bounded the preamble and admission; if it does not, the
614+// guard settles the session, which closes the admitted core service and releases
615+// the listener slot. The bound is derived from the session's own connection-lived
616+// context, so shutdown cancels it too, and it is stopped as soon as Register is
617+// accepted.
618+//
619+// The guard consults the registered marker before settling: a Register whose
620+// handling already began is admitted even when the deadline elapses concurrently
621+// with it, so the boundary between an on-time Register and a silent peer is the
622+// marker, never a timer race.
623+//
624+// The returned stop function is idempotent and joins the guard goroutine, so an
625+// accepted (or settled) session never leaves a timer goroutine behind. A zero
626+// deadline disables the bound; only direct session construction uses that.
627+func (s *serverSession) armRegistrationDeadline() func() {
628+ if s.registerDeadline.IsZero() {
629+ return func() {}
630+ }
631+ ctx, cancel := context.WithDeadline(s.ctx, s.registerDeadline)
632+ stop := make(chan struct{})
633+ done := make(chan struct{})
634+ go func() {
635+ defer close(done)
636+ select {
637+ case <-ctx.Done():
638+ // Only the registration budget itself settles the session; a session
639+ // cancel (shutdown) leaves the terminal cause to the reader. A
640+ // Register already being handled makes the budget moot: the marker is
641+ // set before any connection state is touched, so this check is the
642+ // deadline boundary.
643+ if errors.Is(ctx.Err(), context.DeadlineExceeded) && !s.registered.Load() {
644+ s.abort(errors.Join(ErrRegistrationTimeout, context.DeadlineExceeded))
645+ }
646+ case <-stop:
647+ }
648+ }()
649+ var once sync.Once
650+ return func() {
651+ once.Do(func() {
652+ cancel()
653+ close(stop)
654+ <-done
655+ })
656+ }
657+}
658+
572 // dispatch applies one client frame. A returned error is a protocol or carriage659 // dispatch applies one client frame. A returned error is a protocol or carriage
573 // violation that settles the connection; a refusal the peer can observe is sent660 // violation that settles the connection; a refusal the peer can observe is sent
574 // on the wire and returns nil.661 // on the wire and returns nil.
575 func (s *serverSession) dispatch(message brokerwire.ClientMessage) error {662 func (s *serverSession) dispatch(message brokerwire.ClientMessage) error {
576 switch m := message.(type) {663 switch m := message.(type) {
577 case brokerwire.Register:664 case brokerwire.Register:
665+ // Mark registration as begun before touching connection state, so the
666+ // pre-Register guard sharing the accept-time deadline can distinguish a
667+ // Register arriving at the boundary from a silent peer and never settles
668+ // this healthy connection.
669+ s.registered.Store(true)
578 if err := s.conn.Register(); err != nil {670 if err := s.conn.Register(); err != nil {
579 return errors.Join(ErrProtocol, err)671 return errors.Join(ErrProtocol, err)
580 }672 }
673+ // The client met the accept-time registration budget: stop (and join)
674+ // the pre-Register guard before answering, so this healthy connection is
675+ // no longer bounded by the setup deadline.
676+ s.stopRegistration()
581 return s.send(brokerwire.Registered{Epoch: s.epoch, Connection: s.scope.Connection})677 return s.send(brokerwire.Registered{Epoch: s.epoch, Connection: s.scope.Connection})
582 case brokerwire.Subscribe:678 case brokerwire.Subscribe:
583 if !s.scopeMatches(m.Epoch, m.Connection) {679 if !s.scopeMatches(m.Epoch, m.Connection) {
@@ -771,9 +867,23 @@ func (s *serverSession) Close() error {
771 }867 }
772 868
773 // orderlyDisconnect reports whether one terminal cause is a peer disconnect or a869 // orderlyDisconnect reports whether one terminal cause is a peer disconnect or a
774-// local close rather than a failure.870+// local close rather than a failure. A registration timeout is a peer that
871+// completed admission and then stayed silent: the session settles, but the
872+// outcome is the peer's departure, so a listener draining its sessions never
873+// reports it as its own failure.
874+//
875+// A peer that closes abruptly can surface a reset (or a local-close sentinel
876+// from a concurrently interrupted read) instead of a clean EOF; both are still
877+// the peer's or this side's ordinary departure, never a sandbox failure.
775 func orderlyDisconnect(err error) bool {878 func orderlyDisconnect(err error) bool {
776- return err == nil || errors.Is(err, io.EOF) || errors.Is(err, ErrConnectionClosed) || errors.Is(err, ErrSessionClosed)879+ return err == nil ||
880+ errors.Is(err, io.EOF) ||
881+ errors.Is(err, syscall.ECONNRESET) ||
882+ errors.Is(err, net.ErrClosed) ||
883+ errors.Is(err, fs.ErrClosed) ||
884+ errors.Is(err, ErrConnectionClosed) ||
885+ errors.Is(err, ErrSessionClosed) ||
886+ errors.Is(err, ErrRegistrationTimeout)
777 }887 }
778 888
779 // ConnectionID returns the identity assigned to this connection at accept.889 // ConnectionID returns the identity assigned to this connection at accept.
M internal/adapters/brokeripc/session_shutdown_test.go +1 -1
@@ -84,7 +84,7 @@ func TestProtocolViolationWithParkedStreamWriterReleasesSession(t *testing.T) {
84 require.True(t, ok, "the IPC carriage must be a bounded transport")84 require.True(t, ok, "the IPC carriage must be a bounded transport")
85 core := newTestCore(epoch)85 core := newTestCore(epoch)
86 released := make(chan struct{}, 1)86 released := make(chan struct{}, 1)
87- session, err := newServerSession(epoch, transport, brokerwire.DefaultCeilings(), core, Config{}, func() { released <- struct{}{} })87+ session, err := newServerSession(epoch, transport, brokerwire.DefaultCeilings(), core, Config{}, func() { released <- struct{}{} }, time.Time{})
88 require.NoError(t, err)88 require.NoError(t, err)
89 89
90 // The peer's Register already happened; the duplicate Register below is the90 // The peer's Register already happened; the duplicate Register below is the
M internal/adapters/brokeripc/session_test.go +3 -3
@@ -30,7 +30,7 @@ func TestResubscribePublishesCurrentSnapshotPerGeneration(t *testing.T) {
30 for burst := 0; burst < bursts; burst++ {30 for burst := 0; burst < bursts; burst++ {
31 transport := newRecordingTransport()31 transport := newRecordingTransport()
32 core := newTestCore(epoch)32 core := newTestCore(epoch)
33- session, err := newServerSession(epoch, transport, brokerwire.DefaultCeilings(), core, Config{}, func() {})33+ session, err := newServerSession(epoch, transport, brokerwire.DefaultCeilings(), core, Config{}, func() {}, time.Time{})
34 require.NoError(t, err)34 require.NoError(t, err)
35 require.NoError(t, session.conn.Register())35 require.NoError(t, session.conn.Register())
36 36
@@ -71,7 +71,7 @@ func TestSupersededPublisherOwnsItsWake(t *testing.T) {
71 const epoch = ports.BrokerEpoch(0x52)71 const epoch = ports.BrokerEpoch(0x52)
72 transport := newRecordingTransport()72 transport := newRecordingTransport()
73 core := newTestCore(epoch)73 core := newTestCore(epoch)
74- session, err := newServerSession(epoch, transport, brokerwire.DefaultCeilings(), core, Config{}, func() {})74+ session, err := newServerSession(epoch, transport, brokerwire.DefaultCeilings(), core, Config{}, func() {}, time.Time{})
75 require.NoError(t, err)75 require.NoError(t, err)
76 require.NoError(t, session.conn.Register())76 require.NoError(t, session.conn.Register())
77 t.Cleanup(func() { session.shutdown() })77 t.Cleanup(func() { session.shutdown() })
@@ -103,7 +103,7 @@ func TestSessionCloseStreamFillsConnectionScope(t *testing.T) {
103 const epoch = ports.BrokerEpoch(0x53)103 const epoch = ports.BrokerEpoch(0x53)
104 transport := newRecordingTransport()104 transport := newRecordingTransport()
105 core := newTestCore(epoch)105 core := newTestCore(epoch)
106- session, err := newServerSession(epoch, transport, brokerwire.DefaultCeilings(), core, Config{}, func() {})106+ session, err := newServerSession(epoch, transport, brokerwire.DefaultCeilings(), core, Config{}, func() {}, time.Time{})
107 require.NoError(t, err)107 require.NoError(t, err)
108 t.Cleanup(func() { session.shutdown() })108 t.Cleanup(func() { session.shutdown() })
109 109
M internal/adapters/daemonmux/pump_test.go +48 -9
@@ -197,6 +197,43 @@ func requireEngineEventually(t *testing.T, pump *Pump, cond func(*StreamEngine)
197 require.Eventually(t, func() bool { return cond(pump.Engine()) }, 5*time.Second, time.Millisecond, msgAndArgs...)197 require.Eventually(t, func() bool { return cond(pump.Engine()) }, 5*time.Second, time.Millisecond, msgAndArgs...)
198 }198 }
199 199
200+// requireStreamTerminalEventually waits until one stream has reached its
201+// terminal state with Done closed, then returns that settled read.
202+func requireStreamTerminalEventually(t *testing.T, pump *Pump, id PhysicalStreamID) StreamStatus {
203+ t.Helper()
204+ var status StreamStatus
205+ require.Eventually(t, func() bool {
206+ current, ok := pump.Engine().Status(id)
207+ if !ok {
208+ return false
209+ }
210+ status = current
211+ return current.State == StreamTerminal && channelClosed(current.Done)
212+ }, 5*time.Second, time.Millisecond, "stream %d did not reach terminal", id)
213+ return status
214+}
215+
216+// requireStreamClosingEventually waits until one undrained stream has settled
217+// into closing with its preserved chunk still queued and charged and Done still
218+// open, then returns that settled read. The transition is stable until the
219+// chunk is drained, so the returned read cannot be a stale intermediate state.
220+func requireStreamClosingEventually(t *testing.T, pump *Pump, id PhysicalStreamID) StreamStatus {
221+ t.Helper()
222+ var status StreamStatus
223+ require.Eventually(t, func() bool {
224+ current, ok := pump.Engine().Status(id)
225+ if !ok {
226+ return false
227+ }
228+ status = current
229+ return current.State == StreamClosing &&
230+ current.QueuedChunks == 1 &&
231+ current.QueuedBytes == 1 &&
232+ !channelClosed(current.Done)
233+ }, 5*time.Second, time.Millisecond, "stream %d did not settle into closing with a preserved chunk", id)
234+ return status
235+}
236+
200 // takeEventually waits until one stream's inbound queue yields a chunk.237 // takeEventually waits until one stream's inbound queue yields a chunk.
201 func takeEventually(t *testing.T, pump *Pump, id PhysicalStreamID) ([]byte, bool) {238 func takeEventually(t *testing.T, pump *Pump, id PhysicalStreamID) ([]byte, bool) {
202 t.Helper()239 t.Helper()
@@ -310,19 +347,21 @@ func TestPumpTwoAndHundredStreams(t *testing.T) {
310 }347 }
311 // The drained stream settles at once; every other stream is closing348 // The drained stream settles at once; every other stream is closing
312 // and still holds its one unread, charged chunk until it is drained.349 // and still holds its one unread, charged chunk until it is drained.
313- requireEngineEventually(t, pump, func(e *StreamEngine) bool { return e.Live() == count-1 })350+ // The reader applies the closes asynchronously and in order, so each
314- require.Equal(t, count-1, pump.Engine().AggregateBytes())351+ // stream is observed on its own eventual transition: a global Live
315- require.False(t, pump.Engine().Closed())352+ // precondition would pass as soon as the drained stream settles,
316- require.False(t, channelClosed(pump.Done()))353+ // before a later stream has applied its own close.
317- terminal := mustStatus(t, pump.Engine(), ids[0])354+ terminal := requireStreamTerminalEventually(t, pump, ids[0])
318 require.Equal(t, StreamTerminal, terminal.State)355 require.Equal(t, StreamTerminal, terminal.State)
319- require.True(t, channelClosed(terminal.Done))
320 for _, id := range ids[1:] {356 for _, id := range ids[1:] {
321- status := mustStatus(t, pump.Engine(), id)357+ status := requireStreamClosingEventually(t, pump, id)
322- require.Equal(t, StreamClosing, status.State)
323- require.False(t, channelClosed(status.Done))
324 require.Equal(t, 1, status.QueuedChunks)358 require.Equal(t, 1, status.QueuedChunks)
359+ require.Equal(t, 1, status.QueuedBytes)
325 }360 }
361+ require.Equal(t, count-1, pump.Engine().Live())
362+ require.Equal(t, count-1, pump.Engine().AggregateBytes())
363+ require.False(t, pump.Engine().Closed())
364+ require.False(t, channelClosed(pump.Done()))
326 365
327 // Draining every preserved chunk releases its bytes and terminalizes366 // Draining every preserved chunk releases its bytes and terminalizes
328 // the stream; the physical connection stays open.367 // the stream; the physical connection stays open.
M internal/adapters/lifecycle/flock.go +52 -2
@@ -40,9 +40,9 @@ func TryAcquire(runtimeDir string) (*Owner, error) {
40 }40 }
41 41
42 path := Path(runtimeDir)42 path := Path(runtimeDir)
43- file, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR, 0o600)43+ file, err := openLockFile(path)
44 if err != nil {44 if err != nil {
45- return nil, fmt.Errorf("lifecycle: opening %s: %w", path, err)45+ return nil, err
46 }46 }
47 if err := syscall.Flock(int(file.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil {47 if err := syscall.Flock(int(file.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil {
48 closeErr := file.Close()48 closeErr := file.Close()
@@ -54,6 +54,56 @@ func TryAcquire(runtimeDir string) (*Owner, error) {
54 return &Owner{file: file, path: path}, nil54 return &Owner{file: file, path: path}, nil
55 }55 }
56 56
57+// openLockFile opens the lock inode without following a symlink and refuses to
58+// trust anything that is not a regular, owner-only file.
59+//
60+// O_NOFOLLOW makes a symlinked lock path fail closed with ELOOP instead of
61+// flocking the link target, and the fstat check fails closed on a directory,
62+// FIFO, device, or a file with group/other access, so a path another user could
63+// plant can never become the lifecycle lock. Only group and other access is
64+// refused: an owner-only file is trusted regardless of the exact owner bit
65+// pattern, so a lock created under a restrictive umask (or normalized by a
66+// different tool) is never a permanent, unrecoverable startup failure. A newly
67+// created lock is additionally chmodded to 0600 so a permissive umask can never
68+// leave group/other bits behind. The failure is reported as an ordinary error,
69+// never ErrBusy.
70+func openLockFile(path string) (*os.File, error) {
71+ fd, err := syscall.Open(path, syscall.O_CREAT|syscall.O_EXCL|syscall.O_RDWR|syscall.O_CLOEXEC|syscall.O_NOFOLLOW, 0o600)
72+ created := err == nil
73+ if err != nil {
74+ if !errors.Is(err, syscall.EEXIST) {
75+ if errors.Is(err, syscall.ELOOP) {
76+ return nil, fmt.Errorf("lifecycle: %s is a symlink", path)
77+ }
78+ return nil, fmt.Errorf("lifecycle: opening %s: %w", path, err)
79+ }
80+ fd, err = syscall.Open(path, syscall.O_RDWR|syscall.O_CLOEXEC|syscall.O_NOFOLLOW, 0)
81+ if err != nil {
82+ if errors.Is(err, syscall.ELOOP) {
83+ return nil, fmt.Errorf("lifecycle: %s is a symlink", path)
84+ }
85+ return nil, fmt.Errorf("lifecycle: opening %s: %w", path, err)
86+ }
87+ }
88+ file := os.NewFile(uintptr(fd), path)
89+ if created {
90+ if err := syscall.Fchmod(fd, 0o600); err != nil {
91+ return nil, errors.Join(fmt.Errorf("lifecycle: chmod %s: %w", path, err), file.Close())
92+ }
93+ }
94+ info, err := file.Stat()
95+ if err != nil {
96+ return nil, errors.Join(fmt.Errorf("lifecycle: stat %s: %w", path, err), file.Close())
97+ }
98+ if !info.Mode().IsRegular() {
99+ return nil, errors.Join(fmt.Errorf("lifecycle: %s is not a regular file", path), file.Close())
100+ }
101+ if info.Mode().Perm()&0o077 != 0 {
102+ return nil, errors.Join(fmt.Errorf("lifecycle: %s has permissions %04o, want owner-only", path, info.Mode().Perm()), file.Close())
103+ }
104+ return file, nil
105+}
106+
57 // Acquire waits until lifecycle ownership is available or ctx is cancelled.107 // Acquire waits until lifecycle ownership is available or ctx is cancelled.
58 // Only lock contention is retried; filesystem and lock errors fail closed.108 // Only lock contention is retried; filesystem and lock errors fail closed.
59 func Acquire(ctx context.Context, runtimeDir string, retry time.Duration) (*Owner, error) {109 func Acquire(ctx context.Context, runtimeDir string, retry time.Duration) (*Owner, error) {
M internal/adapters/lifecycle/flock_test.go +87 -0
@@ -11,6 +11,7 @@ import (
11 "os"11 "os"
12 "os/exec"12 "os/exec"
13 "path/filepath"13 "path/filepath"
14+ "syscall"
14 "testing"15 "testing"
15 "time"16 "time"
16 17
@@ -36,6 +37,92 @@ func TestLifecycleLock(t *testing.T) {
36 t.Run("process contention and reacquire", testLifecycleLockProcessContentionAndReacquire)37 t.Run("process contention and reacquire", testLifecycleLockProcessContentionAndReacquire)
37 t.Run("acquire cancellation", testLifecycleLockAcquireCancellation)38 t.Run("acquire cancellation", testLifecycleLockAcquireCancellation)
38 t.Run("unavailable runtime directory", testLifecycleLockUnavailableRuntimeDirectory)39 t.Run("unavailable runtime directory", testLifecycleLockUnavailableRuntimeDirectory)
40+ t.Run("lock file is owner-only regular", testLifecycleLockIsOwnerOnlyRegular)
41+ t.Run("owner-only variant is accepted", testLifecycleLockOwnerOnlyVariantAccepted)
42+ t.Run("restrictive umask normalizes to owner-only", testLifecycleLockRestrictiveUmaskNormalizes)
43+ t.Run("symlinked lock path fails closed", testLifecycleLockSymlinkFailsClosed)
44+ t.Run("nonregular lock path fails closed", testLifecycleLockNonRegularFailsClosed)
45+ t.Run("loose permissions fail closed", testLifecycleLockLoosePermissionsFailClosed)
46+}
47+
48+func testLifecycleLockIsOwnerOnlyRegular(t *testing.T) {
49+ runtimeDir := filepath.Join(t.TempDir(), "runtime")
50+ owner, err := TryAcquire(runtimeDir)
51+ require.NoError(t, err)
52+ defer func() { require.NoError(t, owner.Release()) }()
53+
54+ info, err := os.Lstat(Path(runtimeDir))
55+ require.NoError(t, err)
56+ require.True(t, info.Mode().IsRegular(), "the lock must be a regular file")
57+ require.Equal(t, os.FileMode(0o600), info.Mode().Perm(), "the lock must be owner-only")
58+}
59+
60+// testLifecycleLockOwnerOnlyVariantAccepted proves a pre-existing owner-only
61+// lock with a different owner bit pattern is trusted, so an installation whose
62+// lock was created under a variant umask can still start the daemon.
63+func testLifecycleLockOwnerOnlyVariantAccepted(t *testing.T) {
64+ runtimeDir := filepath.Join(t.TempDir(), "runtime")
65+ require.NoError(t, os.MkdirAll(runtimeDir, 0o700))
66+ require.NoError(t, os.WriteFile(Path(runtimeDir), nil, 0o600))
67+ require.NoError(t, os.Chmod(Path(runtimeDir), 0o700))
68+
69+ owner, err := TryAcquire(runtimeDir)
70+ require.NoError(t, err)
71+ require.NoError(t, owner.Release())
72+}
73+
74+// testLifecycleLockRestrictiveUmaskNormalizes proves a newly created lock is
75+// chmodded to 0600 even when the process umask would have stripped owner bits,
76+// so the lock is always exactly owner-only after creation.
77+func testLifecycleLockRestrictiveUmaskNormalizes(t *testing.T) {
78+ runtimeDir := filepath.Join(t.TempDir(), "runtime")
79+ require.NoError(t, os.MkdirAll(runtimeDir, 0o700))
80+ previous := syscall.Umask(0o400)
81+ defer syscall.Umask(previous)
82+
83+ owner, err := TryAcquire(runtimeDir)
84+ require.NoError(t, err)
85+ defer func() { require.NoError(t, owner.Release()) }()
86+
87+ info, err := os.Lstat(Path(runtimeDir))
88+ require.NoError(t, err)
89+ require.Equal(t, os.FileMode(0o600), info.Mode().Perm(), "a newly created lock must be normalized to owner-only")
90+}
91+
92+func testLifecycleLockSymlinkFailsClosed(t *testing.T) {
93+ runtimeDir := filepath.Join(t.TempDir(), "runtime")
94+ require.NoError(t, os.MkdirAll(runtimeDir, 0o700))
95+ target := filepath.Join(t.TempDir(), "target.lock")
96+ require.NoError(t, os.WriteFile(target, nil, 0o600))
97+ require.NoError(t, os.Symlink(target, Path(runtimeDir)))
98+
99+ _, err := TryAcquire(runtimeDir)
100+ require.Error(t, err)
101+ require.False(t, errors.Is(err, ErrBusy), "a symlinked lock path is an error, not contention")
102+ require.Contains(t, err.Error(), "symlink")
103+}
104+
105+func testLifecycleLockNonRegularFailsClosed(t *testing.T) {
106+ runtimeDir := filepath.Join(t.TempDir(), "runtime")
107+ require.NoError(t, os.MkdirAll(runtimeDir, 0o700))
108+ require.NoError(t, syscall.Mkfifo(Path(runtimeDir), 0o600))
109+
110+ _, err := TryAcquire(runtimeDir)
111+ require.Error(t, err)
112+ require.False(t, errors.Is(err, ErrBusy), "a non-regular lock path is an error, not contention")
113+ require.Contains(t, err.Error(), "not a regular file")
114+}
115+
116+func testLifecycleLockLoosePermissionsFailClosed(t *testing.T) {
117+ runtimeDir := filepath.Join(t.TempDir(), "runtime")
118+ require.NoError(t, os.MkdirAll(runtimeDir, 0o700))
119+ require.NoError(t, os.WriteFile(Path(runtimeDir), nil, 0o600))
120+ require.NoError(t, os.Chmod(Path(runtimeDir), 0o644))
121+
122+ _, err := TryAcquire(runtimeDir)
123+ require.Error(t, err)
124+ require.False(t, errors.Is(err, ErrBusy), "a loose lock file is an error, not contention")
125+ require.Contains(t, err.Error(), "permissions")
39 }126 }
40 127
41 func testLifecycleLockProcessContentionAndReacquire(t *testing.T) {128 func testLifecycleLockProcessContentionAndReacquire(t *testing.T) {
M internal/adapters/sshstdio/mux.go +37 -0
@@ -45,6 +45,7 @@ import (
45 "os"45 "os"
46 "os/exec"46 "os/exec"
47 "strings"47 "strings"
48+ "sync"
48 "unicode"49 "unicode"
49 "unicode/utf8"50 "unicode/utf8"
50 51
@@ -99,6 +100,34 @@ func DialMuxContext(ctx context.Context, spec CommandSpec, logger *slog.Logger,
99 return dialMuxContext(ctx, spec, logger, opts...)100 return dialMuxContext(ctx, spec, logger, opts...)
100 }101 }
101 102
103+// NewDiagnosticSink returns a bounded stderr sink that captures at most limit
104+// bytes and reports a sanitized, truncated diagnostic: terminal escapes and
105+// control bytes are stripped, so a hostile peer can neither grow the capture
106+// without bound nor inject control bytes into a log. It is the same bounded,
107+// sanitized capture the mux carriage uses for its own child diagnostics, shared
108+// with callers (such as the broker's authenticated QUIC bootstrap) that start
109+// an ssh child of their own and must never surface raw remote stderr.
110+func NewDiagnosticSink(limit int) *DiagnosticSink {
111+ if limit <= 0 {
112+ limit = muxStderrLimit
113+ }
114+ return &DiagnosticSink{capped: newCappedDiagnostic(limit)}
115+}
116+
117+// DiagnosticSink is a bounded, sanitizing, concurrent-safe io.Writer for one
118+// child's stderr. It is safe for the os/exec stderr copy goroutine to write
119+// while another goroutine reads String: the read is internally synchronized and
120+// observes a bounded, sanitized snapshot rather than a partial capture.
121+type DiagnosticSink struct{ capped *cappedDiagnostic }
122+
123+// Write captures at most the sink's ceiling and records that the rest was
124+// dropped, so the child is never blocked writing diagnostics.
125+func (s *DiagnosticSink) Write(p []byte) (int, error) { return s.capped.Write(p) }
126+
127+// String returns the captured text sanitized and truncated to the sink's
128+// ceiling, so every reader observes a bounded, control-byte-free value.
129+func (s *DiagnosticSink) String() string { return s.capped.String() }
130+
102 // NewStdioTransport returns the helper half of one SSH stdio carriage: the131 // NewStdioTransport returns the helper half of one SSH stdio carriage: the
103 // current process' own stdin and stdout framed as the raw bounded transport the132 // current process' own stdin and stdout framed as the raw bounded transport the
104 // daemonmux daemon path consumes. It is the counterpart of DialMuxContext, so a133 // daemonmux daemon path consumes. It is the counterpart of DialMuxContext, so a
@@ -157,6 +186,10 @@ func dialMuxContext(ctx context.Context, spec CommandSpec, logger *slog.Logger,
157 // non-printable bytes are stripped, so a hostile peer can neither grow the186 // non-printable bytes are stripped, so a hostile peer can neither grow the
158 // capture without bound nor inject control bytes into a log.187 // capture without bound nor inject control bytes into a log.
159 type cappedDiagnostic struct {188 type cappedDiagnostic struct {
189+ // mu makes Write and String safe to call concurrently: an os/exec stderr
190+ // copy goroutine may still be writing while a reader renders the diagnostic,
191+ // and the capture must never be observed partially written.
192+ mu sync.Mutex
160 buf bytes.Buffer193 buf bytes.Buffer
161 limit int194 limit int
162 overflow bool195 overflow bool
@@ -169,6 +202,8 @@ func newCappedDiagnostic(limit int) *cappedDiagnostic {
169 // Write captures at most the configured ceiling and records that the rest was202 // Write captures at most the configured ceiling and records that the rest was
170 // dropped, so the child is never blocked writing diagnostics.203 // dropped, so the child is never blocked writing diagnostics.
171 func (d *cappedDiagnostic) Write(p []byte) (int, error) {204 func (d *cappedDiagnostic) Write(p []byte) (int, error) {
205+ d.mu.Lock()
206+ defer d.mu.Unlock()
172 remaining := d.limit - d.buf.Len()207 remaining := d.limit - d.buf.Len()
173 if remaining <= 0 {208 if remaining <= 0 {
174 d.overflow = true209 d.overflow = true
@@ -185,6 +220,8 @@ func (d *cappedDiagnostic) Write(p []byte) (int, error) {
185 // String returns the captured text sanitized and truncated to the diagnostic220 // String returns the captured text sanitized and truncated to the diagnostic
186 // ceiling, so every reader observes a bounded, control-byte-free value.221 // ceiling, so every reader observes a bounded, control-byte-free value.
187 func (d *cappedDiagnostic) String() string {222 func (d *cappedDiagnostic) String() string {
223+ d.mu.Lock()
224+ defer d.mu.Unlock()
188 return sanitizeDiagnostic(d.buf.String())225 return sanitizeDiagnostic(d.buf.String())
189 }226 }
190 227
A internal/app/broker_offline.go +376 -0
@@ -0,0 +1,376 @@
1+package app
2+
3+import (
4+ "context"
5+ "crypto/rand"
6+ "encoding/binary"
7+ "errors"
8+ "fmt"
9+ "io"
10+ "log/slog"
11+ "os/signal"
12+ "strings"
13+ "syscall"
14+ "time"
15+
16+ "github.com/bnema/vev/internal/adapters/brokerconfig"
17+ "github.com/bnema/vev/internal/adapters/brokeripc"
18+ "github.com/bnema/vev/internal/adapters/brokerstore"
19+ "github.com/bnema/vev/internal/adapters/clock"
20+ "github.com/bnema/vev/internal/adapters/ipc"
21+ "github.com/bnema/vev/internal/adapters/lifecycle"
22+ "github.com/bnema/vev/internal/logging"
23+ "github.com/bnema/vev/internal/platform"
24+ "github.com/bnema/vev/internal/ports"
25+ "github.com/bnema/vev/internal/usecase/broker"
26+ "github.com/bnema/vev/pkg/safedir"
27+)
28+
29+// Offline broker foreground composition (Plan 001 P3.4 slice C).
30+//
31+// `_broker-serve` is a hidden, isolated sandbox command. It takes one strict
32+// offline root, derives only sandbox runtime, state, and log paths from it, and
33+// composes the P3.4 broker core and P3.3 local IPC listener over the immutable
34+// config marker. It never reads or writes production runtime, state, or config
35+// and never changes an ordinary command, path, or factory.
36+//
37+// Nothing here spawns a detached process, connects or spawns a daemon, reports
38+// status, or reaches SSH or QUIC: those remain later slices. The composed
39+// process is foreground: it owns one lifetime lock for its whole run, serves
40+// until a signal or its parent context ends it, or until the broker idles out,
41+// and then shuts down in a fixed order.
42+
43+const brokerServeCommand = "_broker-serve"
44+
45+// brokerServePoolLimits bound the offline broker pool. They are explicit, not
46+// derived from the environment, and small enough that a stalled peer is
47+// refused rather than absorbed.
48+var brokerServePoolLimits = broker.PoolLimits{
49+ Physical: 64,
50+ Clients: brokeripc.DefaultMaxClients,
51+ Streams: 256,
52+ StreamsPerClient: 64,
53+ Idle: 5 * time.Minute,
54+}
55+
56+// brokerServeOptions is the parsed hidden invocation.
57+type brokerServeOptions struct {
58+ offlineRoot string
59+ // idleGrace is zero when the operator did not override it; the supervisor
60+ // then applies its own 5m default.
61+ idleGrace time.Duration
62+}
63+
64+// parseBrokerServeArgs strictly parses `_broker-serve --offline-root ABS
65+// [--idle-grace DURATION]`. Unknown flags, duplicate flags, positionals, a
66+// missing value, and a non-positive or unparsable duration are refused; the
67+// path itself is validated later against the filesystem.
68+func parseBrokerServeArgs(args []string) (command, error) {
69+ if len(args) == 0 {
70+ return command{}, usagef("`%s` requires --offline-root", brokerServeCommand)
71+ }
72+ options := brokerServeOptions{}
73+ var rootSet, graceSet bool
74+ for index := 0; index < len(args); {
75+ switch args[index] {
76+ case "--offline-root":
77+ if rootSet {
78+ return command{}, usagef("`%s` received duplicate --offline-root", brokerServeCommand)
79+ }
80+ if index+1 >= len(args) || args[index+1] == "" {
81+ return command{}, usagef("`--offline-root` requires a path")
82+ }
83+ options.offlineRoot = args[index+1]
84+ rootSet = true
85+ index += 2
86+ case "--idle-grace":
87+ if graceSet {
88+ return command{}, usagef("`%s` received duplicate --idle-grace", brokerServeCommand)
89+ }
90+ if index+1 >= len(args) {
91+ return command{}, usagef("`--idle-grace` requires a duration")
92+ }
93+ grace, err := time.ParseDuration(args[index+1])
94+ if err != nil {
95+ return command{}, usagef("`--idle-grace` %q is not a duration", args[index+1])
96+ }
97+ if grace <= 0 {
98+ return command{}, usagef("`--idle-grace` must be positive")
99+ }
100+ options.idleGrace = grace
101+ graceSet = true
102+ index += 2
103+ default:
104+ if strings.HasPrefix(args[index], "-") {
105+ return command{}, usagef("unknown flag %q for `%s`", args[index], brokerServeCommand)
106+ }
107+ return command{}, usagef("`%s` does not accept positional arguments", brokerServeCommand)
108+ }
109+ }
110+ if !rootSet {
111+ return command{}, usagef("`%s` requires --offline-root", brokerServeCommand)
112+ }
113+ return command{kind: kindBrokerServe, brokerServe: options}, nil
114+}
115+
116+// brokerServeDeps are the composition seams. Production supplies the real
117+// adapters; tests override the clock, the owner lock, the logger, and the
118+// readiness hook so idle shutdown and cleanup are deterministic.
119+type brokerServeDeps struct {
120+ clock func() ports.Clock
121+ acquire func(string) (lifecycleOwnership, error)
122+ newLogger func(string) (*slog.Logger, io.Closer, error)
123+ newEpoch func() (ports.BrokerEpoch, error)
124+ listen func(string, ports.BrokerEpoch, ports.BrokerAuthority, brokeripc.Config) (ports.BrokerListener, error)
125+ onReady func(socketPath string)
126+}
127+
128+// defaultBrokerServeDeps builds the production seams.
129+func defaultBrokerServeDeps() brokerServeDeps {
130+ return brokerServeDeps{
131+ clock: func() ports.Clock { return clock.New() },
132+ acquire: func(runtimeDir string) (lifecycleOwnership, error) {
133+ return lifecycle.TryAcquire(runtimeDir)
134+ },
135+ newLogger: sandboxLogging,
136+ newEpoch: newBrokerEpoch,
137+ listen: func(path string, epoch ports.BrokerEpoch, authority ports.BrokerAuthority, cfg brokeripc.Config) (ports.BrokerListener, error) {
138+ return brokeripc.Listen(path, epoch, authority, cfg)
139+ },
140+ }
141+}
142+
143+// runBrokerServeCommand enters the hidden foreground process: it installs
144+// SIGINT/SIGTERM cancellation over the parent context and runs the sandbox.
145+func runBrokerServeCommand(ctx context.Context, options brokerServeOptions) error {
146+ ctx, stop := signal.NotifyContext(ctx, syscall.SIGINT, syscall.SIGTERM)
147+ defer stop()
148+ return runBrokerServe(ctx, options, defaultBrokerServeDeps())
149+}
150+
151+// sandboxLogging configures broker logging inside the sandbox log directory.
152+// It never consults the production state directory.
153+func sandboxLogging(dir string) (*slog.Logger, io.Closer, error) {
154+ return logging.Setup(logging.Config{
155+ Dir: dir,
156+ Component: logging.Daemon,
157+ Level: logging.EnvLevel(),
158+ MaxBytes: logging.DefaultMaxBytes,
159+ })
160+}
161+
162+// newBrokerEpoch samples one cryptographically random non-zero broker epoch.
163+func newBrokerEpoch() (ports.BrokerEpoch, error) {
164+ var raw [8]byte
165+ if _, err := rand.Read(raw[:]); err != nil {
166+ return 0, fmt.Errorf("vev: sample broker epoch: %w", err)
167+ }
168+ epoch := ports.BrokerEpoch(binary.BigEndian.Uint64(raw[:]))
169+ if epoch == 0 {
170+ epoch = 1
171+ }
172+ return epoch, nil
173+}
174+
175+// offlineLayout resolves the hidden offline sandbox layout for one operator
176+// root, refusing any root that overlaps the production runtime or state paths.
177+func offlineLayout(offlineRoot string) (brokerconfig.Layout, error) {
178+ return brokerconfig.ResolveLayout(offlineRoot, []string{ipc.SocketDir(), platform.StateDir()})
179+}
180+
181+// effectiveIdleGrace resolves the sandbox idle grace from an explicit flag and
182+// the provisioned configuration. The provisioned value is the single
183+// deterministic source: a status probe cannot read a running broker's effective
184+// grace through the broker IPC protocol, so an explicit grace that conflicts
185+// with a provisioned one is diagnosed here rather than silently applied.
186+func effectiveIdleGrace(config *brokerconfig.Config, explicit time.Duration) (time.Duration, error) {
187+ provisioned, ok := config.IdleGrace()
188+ switch {
189+ case explicit <= 0:
190+ if ok {
191+ return provisioned, nil
192+ }
193+ // Zero lets the supervisor apply its own built-in default.
194+ return 0, nil
195+ case ok && explicit != provisioned:
196+ return 0, usagef("`--idle-grace` %s conflicts with the provisioned idle grace %s", explicit, provisioned)
197+ default:
198+ return explicit, nil
199+ }
200+}
201+
202+// runBrokerServe runs one isolated offline broker in the foreground until ctx
203+// is cancelled or the broker idles out, then shuts down in the fixed order
204+// listener -> pool -> registry -> store -> log -> owner lock.
205+//
206+// Every resource is acquired in dependency order and released in reverse on
207+// any failure, so a partial setup never leaks a store lock, an owner lock, a
208+// bound socket, or a supervisor goroutine. The temporary startup lease pins the
209+// supervisor open from construction until the listener and its accept drain are
210+// ready, so the idle timer can never fire during setup.
211+func runBrokerServe(ctx context.Context, options brokerServeOptions, deps brokerServeDeps) (retErr error) {
212+ layout, err := offlineLayout(options.offlineRoot)
213+ if err != nil {
214+ return err
215+ }
216+ for _, dir := range []string{layout.Root, layout.Runtime, layout.State, layout.Log} {
217+ if err := safedir.EnsurePrivate(dir); err != nil {
218+ return fmt.Errorf("vev: secure broker sandbox directory %s: %w", dir, err)
219+ }
220+ }
221+ // ResolveLayout validates the root textually and stops at the first
222+ // component that does not exist yet; now that EnsurePrivate has created the
223+ // sandbox paths, re-walk every component so a component swapped for a
224+ // symlink in that window fails closed instead of being followed.
225+ if err := layout.VerifyCreated(); err != nil {
226+ return fmt.Errorf("vev: verify broker sandbox paths: %w", err)
227+ }
228+ config, err := brokerconfig.Load(layout)
229+ if err != nil {
230+ return err
231+ }
232+ idleGrace, err := effectiveIdleGrace(config, options.idleGrace)
233+ if err != nil {
234+ return err
235+ }
236+ owner, err := deps.acquire(layout.Runtime)
237+ if err != nil {
238+ return fmt.Errorf("vev: broker sandbox ownership: %w", err)
239+ }
240+ defer func() { retErr = errors.Join(retErr, owner.Release()) }()
241+
242+ log, logCloser, err := deps.newLogger(layout.Log)
243+ if err != nil {
244+ return err
245+ }
246+ defer func() { retErr = errors.Join(retErr, logCloser.Close()) }()
247+
248+ store, err := brokerstore.OpenOffline(brokerstore.Options{Dir: layout.State})
249+ if err != nil {
250+ return fmt.Errorf("vev: open broker sandbox store: %w", err)
251+ }
252+ defer func() { retErr = errors.Join(retErr, store.Close()) }()
253+
254+ epoch, err := deps.newEpoch()
255+ if err != nil {
256+ return err
257+ }
258+ // One clock is shared by the supervisor, registry, and pool so an injected
259+ // clock (for example a manual test clock) governs every timer in the sandbox.
260+ clk := deps.clock()
261+ supervisor, err := broker.NewSupervisor(clk, broker.SupervisorConfig{IdleGrace: idleGrace})
262+ if err != nil {
263+ return err
264+ }
265+ // The supervisor owns the ordered drain; its deferred Close is a safety net
266+ // for every error path below. A successful run closes it explicitly, and the
267+ // flag suppresses the deferred duplicate join so one shutdown error is never
268+ // reported twice.
269+ supervisorClosed := false
270+ defer func() {
271+ if !supervisorClosed {
272+ retErr = errors.Join(retErr, supervisor.Close())
273+ }
274+ }()
275+
276+ // The startup lease pins the broker open until the listener and its accept
277+ // drain are ready. It is released exactly once, and again on any error path.
278+ startupLease, err := supervisor.AdmitOperation()
279+ if err != nil {
280+ return err
281+ }
282+ defer startupLease.Release()
283+
284+ registry, err := broker.NewRegistryWithConfig(epoch, store, nil, clk, log, broker.RegistryConfig{ObservationDisabled: true})
285+ if err != nil {
286+ return err
287+ }
288+ // Registration order is shutdown order reversed: registering the registry
289+ // first, then the pool, then the listener drains them listener -> pool ->
290+ // registry.
291+ if err := supervisor.RegisterRunner("registry", registry); err != nil {
292+ return err
293+ }
294+ connector, err := brokerMuxConnector(config, log)
295+ if err != nil {
296+ return err
297+ }
298+ pool, err := broker.NewPool(epoch, config.Resolver(), connector, clk, brokerServePoolLimits)
299+ if err != nil {
300+ return err
301+ }
302+ if err := supervisor.RegisterCloseable("pool", pool); err != nil {
303+ return err
304+ }
305+ authority, err := broker.NewAuthority(epoch, registry, pool, supervisor)
306+ if err != nil {
307+ return err
308+ }
309+ socketPath := brokeripc.SocketPath(layout.Runtime)
310+ listener, err := deps.listen(socketPath, epoch, authority, brokeripc.Config{})
311+ if err != nil {
312+ return fmt.Errorf("vev: listen on broker sandbox socket: %w", err)
313+ }
314+ if err := supervisor.RegisterCloseable("listener", listener); err != nil {
315+ _ = listener.Close()
316+ return err
317+ }
318+
319+ started := make(chan struct{})
320+ drained := make(chan struct{})
321+ failed := make(chan struct{})
322+ go drainBrokerAccept(listener, log, started, drained, failed)
323+ <-started
324+ startupLease.Release()
325+ log.Info("broker_offline_ready", "socket", socketPath, "endpoints", len(config.Endpoints()))
326+ if deps.onReady != nil {
327+ deps.onReady(socketPath)
328+ }
329+
330+ select {
331+ case <-ctx.Done():
332+ case <-supervisor.Done():
333+ case <-failed:
334+ // An unexpected terminal accept failure leaves the socket bound but
335+ // unable to admit work. Commit shutdown so the sandbox tears down in the
336+ // fixed order instead of wedging as a bound, unserving broker.
337+ }
338+ // Closing the supervisor commits shutdown, cancels the registry run, and
339+ // drains the listener, pool, and registry in order. The store, log, and
340+ // owner lock are released by the deferred cleanup afterwards.
341+ closeErr := supervisor.Close()
342+ supervisorClosed = true
343+ <-drained
344+ log.Info("broker_offline_stopped")
345+ return closeErr
346+}
347+
348+// drainBrokerAccept continuously drains admitted broker sessions so the
349+// listener's bounded admission FIFO never stalls.
350+//
351+// A drained session is the listener's own connection-scoped service: closing it
352+// here would end a live client, so it is deliberately left open and owned by
353+// the listener. The drain stops when the listener closes or reports a terminal
354+// accept failure. A listener close is the expected, orderly end of the drain and
355+// is never logged; any other terminal accept failure is unexpected: it is logged
356+// as an error and reported on failed so the caller commits shutdown instead of
357+// leaving a bound broker that can no longer admit work.
358+func drainBrokerAccept(listener ports.BrokerListener, log *slog.Logger, started chan<- struct{}, drained chan<- struct{}, failed chan<- struct{}) {
359+ defer close(drained)
360+ close(started)
361+ for {
362+ session, err := listener.Accept()
363+ if err != nil {
364+ if errors.Is(err, brokeripc.ErrListenerClosed) {
365+ return
366+ }
367+ log.Error("broker_offline_accept_failed", "error", err)
368+ close(failed)
369+ return
370+ }
371+ if session == nil {
372+ continue
373+ }
374+ log.Debug("broker_offline_connection", "connection", fmt.Sprintf("%x", session.ConnectionID()))
375+ }
376+}
A internal/app/broker_offline_mux.go +541 -0
@@ -0,0 +1,541 @@
1+package app
2+
3+import (
4+ "context"
5+ "errors"
6+ "fmt"
7+ "io"
8+ "log/slog"
9+ "net"
10+ "os"
11+ "os/exec"
12+ "strconv"
13+ "strings"
14+ "syscall"
15+ "time"
16+
17+ "github.com/bnema/vev/internal/adapters/brokerconfig"
18+ "github.com/bnema/vev/internal/adapters/daemonmux"
19+ "github.com/bnema/vev/internal/adapters/ipc"
20+ "github.com/bnema/vev/internal/adapters/quic"
21+ "github.com/bnema/vev/internal/adapters/sshstdio"
22+ "github.com/bnema/vev/internal/protocol/wire"
23+)
24+
25+// Broker mux remote helper composition (Plan 001 P3.4 slice E).
26+//
27+// The offline broker reaches a remote daemon's private daemonmux carriage
28+// through one of three explicitly provisioned routes: a local Unix mux socket,
29+// an SSH stdio child that runs a hidden mux helper, or an SSH-authenticated
30+// QUIC bootstrap that pins a freshly minted ephemeral certificate. This file
31+// owns both halves.
32+//
33+// The broker side is a transport-selecting daemonmux.EndpointConnector. It
34+// receives only the opaque address the immutable resolver published, looks the
35+// route up in the configuration it was composed with, and dials that route. It
36+// never authenticates the carriage itself: the daemonmux physical preamble
37+// verifies the daemon identity, incarnation, and policy the resolver already
38+// pinned. One caller-supplied setup context bounds bootstrap, pinned dial, and
39+// the daemonmux handshake together; a successful physical connection detaches
40+// from that context and is owned by the pool.
41+//
42+// The remote side is three hidden helpers, excluded from public help:
43+//
44+// - `_broker-mux-stdio --offline-root ABS` bridges its own stdio to the
45+// single provisioned private Unix daemonmux carriage.
46+// - `_broker-mux-quic-bootstrap --offline-root ABS` starts one detached
47+// `_broker-mux-quic-proxy`, forwards its single readiness line, and exits.
48+// - `_broker-mux-quic-proxy --offline-root ABS` mints one ephemeral QUIC
49+// server, admits exactly one authenticated carriage, and bridges it to the
50+// same private Unix daemonmux carriage.
51+//
52+// A helper only ever bridges to the configured private Unix daemonmux
53+// endpoint. It never starts a broker, an observer, or an ordinary daemon, never
54+// dials the production daemon socket, and never fabricates a daemon
55+// incarnation: the daemon on the other side of the carriage owns that value.
56+// The one-time QUIC token and nonce are minted per physical connection, travel
57+// only inside the SSH channel and the encrypted stream, and never reach an
58+// address, a log, or an error. SSH host-key verification, noninteractive
59+// authentication, and the absence of a remote PTY are preserved: the helpers
60+// only add explicit trust inputs that narrow verification, never disable it.
61+
62+const (
63+ // brokerMuxStdioCommand is the hidden remote SSH stdio mux bridge.
64+ brokerMuxStdioCommand = "_broker-mux-stdio"
65+ // brokerMuxQUICBootstrapCommand is the hidden remote QUIC mux bootstrap.
66+ brokerMuxQUICBootstrapCommand = "_broker-mux-quic-bootstrap"
67+ // brokerMuxQUICProxyCommand is the hidden detached remote QUIC mux proxy.
68+ brokerMuxQUICProxyCommand = "_broker-mux-quic-proxy"
69+)
70+
71+// brokerMuxSetupTimeout bounds each side's bootstrap readiness wait. On the
72+// broker side it caps the wait for the local ssh bootstrap child's readiness
73+// line together with that child's kill/reap cleanup; on the remote side it caps
74+// the bootstrap command's wait for the detached proxy's readiness line. It is an
75+// additional backstop only: an earlier caller context deadline always stays
76+// authoritative, because context.WithTimeout keeps the sooner deadline, so the
77+// independent bound can never extend a shorter caller budget.
78+const brokerMuxSetupTimeout = 30 * time.Second
79+
80+// brokerMuxBootstrapReadyTimeout is the effective broker-side bootstrap
81+// readiness budget. Production always uses brokerMuxSetupTimeout; a test
82+// narrows it so a stalled helper that never prints readiness is exercised
83+// deterministically without waiting out the production bound.
84+var brokerMuxBootstrapReadyTimeout = brokerMuxSetupTimeout
85+
86+// brokerMuxQUICDialTimeout bounds the pinned QUIC dial. The setup context is
87+// still authoritative; this narrows the dial itself.
88+const brokerMuxQUICDialTimeout = 15 * time.Second
89+
90+// brokerMuxWaitDelay bounds how long a bootstrap Wait blocks on a captured-
91+// stderr copy goroutine after the direct child exits. A remote helper that
92+// leaves a grandchild holding the inherited stderr pipe can therefore never pin
93+// setup shutdown past this delay; the direct child is still killed and reaped.
94+const brokerMuxWaitDelay = 2 * time.Second
95+
96+// brokerMuxOptions is the parsed hidden mux helper invocation.
97+type brokerMuxOptions struct {
98+ offlineRoot string
99+}
100+
101+// parseBrokerMuxArgs strictly parses `<hidden> --offline-root ABS`. Unknown or
102+// duplicate flags, a missing value, and positionals are refused, exactly like
103+// the foreground sandbox entry point.
104+func parseBrokerMuxArgs(name string, kind cmdKind, args []string) (command, error) {
105+ options := brokerMuxOptions{}
106+ rootSet := false
107+ for index := 0; index < len(args); {
108+ switch args[index] {
109+ case "--offline-root":
110+ if rootSet {
111+ return command{}, usagef("`%s` received duplicate --offline-root", name)
112+ }
113+ if index+1 >= len(args) || args[index+1] == "" {
114+ return command{}, usagef("`--offline-root` requires a path")
115+ }
116+ options.offlineRoot = args[index+1]
117+ rootSet = true
118+ index += 2
119+ default:
120+ if strings.HasPrefix(args[index], "-") {
121+ return command{}, usagef("unknown flag %q for `%s`", args[index], name)
122+ }
123+ return command{}, usagef("`%s` does not accept positional arguments", name)
124+ }
125+ }
126+ if !rootSet {
127+ return command{}, usagef("`%s` requires --offline-root", name)
128+ }
129+ return command{kind: kind, brokerMux: options}, nil
130+}
131+
132+// loadMuxHelperConfig validates one offline root and loads its strict sandbox
133+// configuration. A helper never creates sandbox directories: the root and the
134+// config marker must already exist, so a helper can never provision state.
135+func loadMuxHelperConfig(offlineRoot string) (*brokerconfig.Config, error) {
136+ layout, err := offlineLayout(offlineRoot)
137+ if err != nil {
138+ return nil, err
139+ }
140+ return brokerconfig.Load(layout)
141+}
142+
143+// brokerMuxConnector builds the transport-selecting daemonmux endpoint
144+// connector over one immutable configuration. The dial function receives only
145+// the opaque address the resolver published and refuses any address the
146+// configuration did not produce, so a request can never select a transport or
147+// an address of its own.
148+func brokerMuxConnector(config *brokerconfig.Config, log *slog.Logger) (*daemonmux.EndpointConnector, error) {
149+ if config == nil {
150+ return nil, errors.New("vev: broker mux connector requires a configuration")
151+ }
152+ dial := func(ctx context.Context, address string) (daemonmux.RawFramedTransport, error) {
153+ route, ok := config.RouteByAddress(address)
154+ if !ok {
155+ return nil, fmt.Errorf("vev: broker mux: unknown route address")
156+ }
157+ return dialBrokerRoute(ctx, route, log)
158+ }
159+ return daemonmux.NewEndpointConnector(dial, daemonmux.DefaultMuxCeilings())
160+}
161+
162+// dialBrokerRoute dials one provisioned route as an already-authenticated raw
163+// framed carriage. Authentication, identity, and policy remain the daemonmux
164+// connector's concern; this function only selects the transport.
165+func dialBrokerRoute(ctx context.Context, route brokerconfig.Route, log *slog.Logger) (daemonmux.RawFramedTransport, error) {
166+ switch route.Kind() {
167+ case brokerconfig.RouteUnix:
168+ return ipc.DialMuxContext(ctx, route.Path())
169+ case brokerconfig.RouteSSHStdio:
170+ return sshstdio.DialMuxContext(ctx, sshMuxCommandSpec(route), log)
171+ case brokerconfig.RouteSSHQUIC:
172+ return dialBrokerQUICRoute(ctx, route, log)
173+ default:
174+ return nil, fmt.Errorf("vev: broker mux: unsupported route kind %q", route.Kind())
175+ }
176+}
177+
178+// sshMuxCommandSpec builds the local ssh argv for one provisioned route. It
179+// reuses the mux carriage's own command construction (the option terminator,
180+// the POSIX-quoted remote argv, and `-T` for no remote PTY) and splices in only
181+// the explicit trust inputs the operator provisioned. It never disables
182+// host-key verification, never forces a non-interactive auth bypass, and never
183+// requests a PTY.
184+func sshMuxCommandSpec(route brokerconfig.Route) sshstdio.CommandSpec {
185+ spec := sshstdio.BuildCommandForMux(route.Target(), route.Argv()...)
186+ options := sshTrustOptions(route)
187+ if len(options) == 0 || len(spec.Args) < 2 {
188+ return spec
189+ }
190+ // BuildCommandForMux returns ["-T", "--", target, remote]; insert options
191+ // after `-T` so they stay ssh options and the option terminator still
192+ // protects the target.
193+ args := make([]string, 0, len(spec.Args)+len(options))
194+ args = append(args, spec.Args[0])
195+ args = append(args, options...)
196+ args = append(args, spec.Args[1:]...)
197+ return sshstdio.CommandSpec{Path: spec.Path, Args: args}
198+}
199+
200+// sshTrustOptions renders the provisioned trust inputs as ssh options. A
201+// connect timeout only narrows the attempt; a known-hosts file names the
202+// verification source. So the option carries a single bounded, whitespace-free
203+// value and can never inject a second option.
204+func sshTrustOptions(route brokerconfig.Route) []string {
205+ var options []string
206+ if timeout := route.ConnectTimeout(); timeout > 0 {
207+ seconds := int(timeout / time.Second)
208+ if seconds < 1 {
209+ seconds = 1
210+ }
211+ options = append(options, "-o", fmt.Sprintf("ConnectTimeout=%d", seconds))
212+ }
213+ if knownHosts := route.KnownHostsFile(); knownHosts != "" {
214+ options = append(options, "-o", "UserKnownHostsFile="+knownHosts)
215+ }
216+ return options
217+}
218+
219+// dialBrokerQUICRoute runs one SSH-authenticated QUIC bootstrap and returns the
220+// pinned, authenticated raw bounded carriage. The local ssh child starts the
221+// remote `_broker-mux-quic-bootstrap`, prints exactly one readiness line on
222+// stdout, and exits; the detached remote proxy serves one carriage. A refused
223+// pin, token, or nonce closes the carriage and hands nothing out. Captured
224+// stderr is sanitized and never part of the returned error.
225+//
226+// The caller's setup context bounds the exchange, and brokerMuxSetupTimeout is
227+// an independent backstop: an earlier caller deadline stays authoritative,
228+// while a bootstrap that never prints readiness is still killed and reaped
229+// within the bound and can never pin pool setup. The captured stderr sink is
230+// read only after that kill and reap, and the sink is itself synchronized, so
231+// the os/exec copy goroutine can never race the diagnostic.
232+func dialBrokerQUICRoute(ctx context.Context, route brokerconfig.Route, log *slog.Logger) (wire.BoundedTransport, error) {
233+ spec := sshMuxCommandSpec(route)
234+ stderr := sshstdio.NewDiagnosticSink(0)
235+ cmd := exec.CommandContext(ctx, spec.Path, spec.Args...)
236+ cmd.Stderr = stderr
237+ // A captured stderr is a pipe with a copy goroutine; WaitDelay ensures a
238+ // grandchild that inherits the write end cannot block Wait past the delay
239+ // once the direct child is gone.
240+ cmd.WaitDelay = brokerMuxWaitDelay
241+ stdout, err := cmd.StdoutPipe()
242+ if err != nil {
243+ return nil, brokerQUICUnavailable("bootstrap stdout", err)
244+ }
245+ if err := cmd.Start(); err != nil {
246+ _ = stdout.Close()
247+ return nil, brokerQUICUnavailable("start bootstrap", err)
248+ }
249+ // reapBootstrap kills the local bootstrap and starts a bounded join, exactly
250+ // once. Every error path reaps before it reads the captured stderr sink; the
251+ // sink is safe to read whether or not that bounded join has finished, because
252+ // it synchronizes its own reads and writes. The remote proxy is detached
253+ // in its own session, so killing the local ssh child never ends it.
254+ reaped := false
255+ reapBootstrap := func() {
256+ if reaped {
257+ return
258+ }
259+ reaped = true
260+ reapBootstrapProcess(cmd)
261+ }
262+ defer func() {
263+ reapBootstrap()
264+ _ = stdout.Close()
265+ }()
266+
267+ // The readiness wait is bounded independently of the caller context, so a
268+ // bootstrap that never prints readiness cannot pin pool setup. An earlier
269+ // caller deadline still wins: WithTimeout keeps the sooner deadline.
270+ readyCtx, cancelReadiness := context.WithTimeout(ctx, brokerMuxBootstrapReadyTimeout)
271+ readiness, err := quic.ReadReadiness(readyCtx, stdout)
272+ cancelReadiness()
273+ if err != nil {
274+ reapBootstrap()
275+ logBrokerBootstrapStderr(log, stderr)
276+ return nil, brokerQUICUnavailable("read bootstrap readiness", err)
277+ }
278+ // The bootstrap channel served its purpose: the detached remote proxy owns
279+ // the carriage lifetime. Reap the short-lived local child without killing
280+ // it, so SIGKILL never races the proxy detach. A bounded I/O drain that the
281+ // child's exit outlived (ErrWaitDelay) is not a bootstrap failure.
282+ if waitErr := cmd.Wait(); waitErr != nil && !errors.Is(waitErr, exec.ErrWaitDelay) {
283+ reaped = true
284+ logBrokerBootstrapStderr(log, stderr)
285+ return nil, brokerQUICUnavailable("reap bootstrap", waitErr)
286+ }
287+ reaped = true
288+
289+ // Compose the dial address from the configured ssh target host plus the
290+ // host-independent readiness port: a remote loopback address is never
291+ // dialed. The one-time token and nonce travel only inside the encrypted
292+ // stream after the pinned handshake.
293+ addr, err := brokerQUICPeerAddr(route.Target(), readiness.Port)
294+ if err != nil {
295+ return nil, brokerQUICUnavailable("resolve QUIC peer", err)
296+ }
297+ transport, err := quic.DialMuxContext(ctx, addr, readiness, quic.Config{}, brokerMuxQUICDialTimeout)
298+ if err != nil {
299+ return nil, brokerQUICUnavailable("dial bootstrap QUIC", err)
300+ }
301+ return transport, nil
302+}
303+
304+// brokerQUICPeerAddr composes the dial address from the ssh target host and the
305+// bootstrap-published port, stripping a user prefix, an explicit port, and
306+// bracketed IPv6 literals to the bare host.
307+func brokerQUICPeerAddr(target string, port int) (string, error) {
308+ if port <= 0 || port > 65535 {
309+ return "", fmt.Errorf("invalid bootstrap port %d", port)
310+ }
311+ host := target
312+ if at := strings.LastIndexByte(host, '@'); at >= 0 {
313+ host = host[at+1:]
314+ }
315+ if h, _, err := net.SplitHostPort(host); err == nil {
316+ host = h
317+ }
318+ host = strings.TrimPrefix(strings.TrimSuffix(host, "]"), "[")
319+ if host == "" {
320+ return "", errors.New("empty bootstrap host")
321+ }
322+ return net.JoinHostPort(host, strconv.Itoa(port)), nil
323+}
324+
325+// logBrokerBootstrapStderr records a bounded, sanitized bootstrap diagnostic.
326+// The readiness secret is never on stderr, and no raw remote text is surfaced.
327+// The join performed by the caller is bounded, so the os/exec stderr copy
328+// goroutine may still be writing when this reads the sink; the diagnostic is
329+// nonetheless consistent because DiagnosticSink synchronizes its own reads and
330+// writes and returns a bounded, sanitized snapshot rather than relying on the
331+// join having quiesced the sink.
332+func logBrokerBootstrapStderr(log *slog.Logger, sink *sshstdio.DiagnosticSink) {
333+ if log == nil || sink == nil {
334+ return
335+ }
336+ diagnostic := strings.TrimSpace(sink.String())
337+ if diagnostic == "" {
338+ return
339+ }
340+ log.Warn("broker_mux_bootstrap_stderr", "diagnostic", diagnostic)
341+}
342+
343+// reapBootstrapProcess kills one bootstrap child and starts a bounded join of
344+// its wait. The join is bounded by brokerMuxSetupTimeout so a helper that
345+// leaves a grandchild holding the captured stderr pipe cannot pin cleanup, and
346+// exec.Cmd.WaitDelay narrows the wait much further once the direct child is
347+// gone. When the bound expires the wait goroutine may still be draining the
348+// captured-stderr copy, so a caller that reads the sink afterwards relies on
349+// the sink's own synchronization, not on the join having finished. The direct
350+// child is always killed, so no bootstrap process is ever leaked.
351+func reapBootstrapProcess(cmd *exec.Cmd) {
352+ if cmd == nil || cmd.Process == nil {
353+ return
354+ }
355+ _ = cmd.Process.Kill()
356+ done := make(chan struct{})
357+ go func() {
358+ defer close(done)
359+ _ = cmd.Wait()
360+ }()
361+ timer := time.NewTimer(brokerMuxSetupTimeout)
362+ defer timer.Stop()
363+ select {
364+ case <-done:
365+ case <-timer.C:
366+ }
367+}
368+
369+// brokerQUICUnavailable wraps one bootstrap or pinned-dial failure. The
370+// readiness record is deliberately absent: a returned error never carries the
371+// one-time token, the nonce, or remote stderr.
372+func brokerQUICUnavailable(action string, err error) error {
373+ if err == nil {
374+ err = errors.New("unavailable")
375+ }
376+ return fmt.Errorf("vev: broker mux QUIC unavailable: %s: %w", action, err)
377+}
378+
379+// runBrokerMuxStdioCommand bridges the process' own stdio to the single
380+// provisioned private Unix daemonmux carriage.
381+func runBrokerMuxStdioCommand(ctx context.Context, options brokerMuxOptions) error {
382+ config, err := loadMuxHelperConfig(options.offlineRoot)
383+ if err != nil {
384+ return err
385+ }
386+ route, err := config.LocalMuxRoute()
387+ if err != nil {
388+ return err
389+ }
390+ raw, err := ipc.DialMuxContext(ctx, route.Path())
391+ if err != nil {
392+ return fmt.Errorf("vev: broker mux stdio: dial daemonmux carriage: %w", err)
393+ }
394+ return bridgeMuxTransports(ctx, sshstdio.NewStdioTransport(), raw)
395+}
396+
397+// runBrokerMuxQUICBootstrapCommand starts one detached `_broker-mux-quic-proxy`,
398+// forwards its single readiness line on this process' stdout, and exits. The
399+// proxy owns the carriage lifetime in its own session and survives this short-
400+// lived bootstrap, exactly like the ordinary QUIC bootstrap.
401+func runBrokerMuxQUICBootstrapCommand(ctx context.Context, options brokerMuxOptions) error {
402+ // Validate the sandbox before spawning anything, so a bad root or config
403+ // fails without a readiness line and without a child.
404+ if _, err := loadMuxHelperConfig(options.offlineRoot); err != nil {
405+ return err
406+ }
407+ exe, err := os.Executable()
408+ if err != nil {
409+ return err
410+ }
411+ reader, writer, err := os.Pipe()
412+ if err != nil {
413+ return err
414+ }
415+ defer func() { _ = reader.Close() }()
416+ devNull, err := os.OpenFile(os.DevNull, os.O_RDWR, 0)
417+ if err != nil {
418+ _ = writer.Close()
419+ return err
420+ }
421+ defer func() { _ = devNull.Close() }()
422+
423+ cmd := exec.CommandContext(ctx, exe, brokerMuxQUICProxyCommand, "--offline-root", options.offlineRoot)
424+ // The proxy writes its readiness through the pipe and its diagnostics
425+ // through stderr; stdio is otherwise detached so the SSH bootstrap channel
426+ // can close while the detached proxy owns the carriage.
427+ cmd.Stdin = devNull
428+ cmd.Stdout = writer
429+ cmd.Stderr = devNull
430+ cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
431+ if err := cmd.Start(); err != nil {
432+ _ = writer.Close()
433+ return err
434+ }
435+ _ = writer.Close()
436+
437+ readyCtx, cancel := context.WithTimeout(ctx, brokerMuxSetupTimeout)
438+ defer cancel()
439+ readiness, err := quic.ReadReadiness(readyCtx, reader)
440+ if err != nil {
441+ _ = cmd.Process.Kill()
442+ _ = cmd.Wait()
443+ return fmt.Errorf("vev: broker mux bootstrap readiness: %w", err)
444+ }
445+ line, err := quic.EncodeReadiness(readiness)
446+ if err != nil {
447+ _ = cmd.Process.Kill()
448+ _ = cmd.Wait()
449+ return err
450+ }
451+ if _, err := fmt.Fprintf(os.Stdout, "%s\n", line); err != nil {
452+ _ = cmd.Process.Kill()
453+ _ = cmd.Wait()
454+ return err
455+ }
456+ // Release the detached proxy: it is in its own session and outlives this
457+ // bootstrap. Reaping it here would block until the carriage ends.
458+ return cmd.Process.Release()
459+}
460+
461+// runBrokerMuxQUICProxyCommand mints one ephemeral QUIC server, admits exactly
462+// one authenticated carriage, and bridges it to the single provisioned private
463+// Unix daemonmux carriage. It exits after its carriage ends, on credential
464+// expiry, or on setup failure, so it can never outlive its advertised
465+// credential or serve a second physical connection.
466+func runBrokerMuxQUICProxyCommand(ctx context.Context, options brokerMuxOptions) error {
467+ config, err := loadMuxHelperConfig(options.offlineRoot)
468+ if err != nil {
469+ return err
470+ }
471+ route, err := config.LocalMuxRoute()
472+ if err != nil {
473+ return err
474+ }
475+ server, readiness, err := quic.NewServer()
476+ if err != nil {
477+ return err
478+ }
479+ defer func() { _ = server.Close() }()
480+ line, err := quic.EncodeReadiness(readiness)
481+ if err != nil {
482+ return err
483+ }
484+ if _, err := fmt.Fprintf(os.Stdout, "%s\n", line); err != nil {
485+ return err
486+ }
487+ transport, err := server.AcceptMux(ctx)
488+ if err != nil {
489+ return err
490+ }
491+ defer func() { _ = transport.Close() }()
492+ raw, err := ipc.DialMuxContext(ctx, route.Path())
493+ if err != nil {
494+ return fmt.Errorf("vev: broker mux quic proxy: dial daemonmux carriage: %w", err)
495+ }
496+ return bridgeMuxTransports(ctx, transport, raw)
497+}
498+
499+// bridgeMuxTransports copies raw frames both ways until either carriage ends or
500+// ctx is done, then closes both. An orderly end of either direction (EOF) is a
501+// normal exit for a bridge owner that merely forwards bytes.
502+func bridgeMuxTransports(ctx context.Context, a, b wire.BoundedTransport) error {
503+ ctx, cancel := context.WithCancel(ctx)
504+ defer cancel()
505+ defer func() { _ = a.Close() }()
506+ defer func() { _ = b.Close() }()
507+ errCh := make(chan error, 2)
508+ go func() { errCh <- relayMux(ctx, a, b) }()
509+ go func() { errCh <- relayMux(ctx, b, a) }()
510+ select {
511+ case err := <-errCh:
512+ return orderlyRelay(err)
513+ case <-ctx.Done():
514+ return ctx.Err()
515+ }
516+}
517+
518+// relayMux copies one direction of raw bounded frames.
519+func relayMux(ctx context.Context, src, dst wire.BoundedTransport) error {
520+ for {
521+ envelope, err := src.RecvBounded(wire.AbsoluteEnvelopeLimit)
522+ if err != nil {
523+ return err
524+ }
525+ if err := dst.Send(envelope); err != nil {
526+ return err
527+ }
528+ if err := ctx.Err(); err != nil {
529+ return err
530+ }
531+ }
532+}
533+
534+// orderlyRelay treats EOF as a clean bridge end and passes every other outcome
535+// through unchanged.
536+func orderlyRelay(err error) error {
537+ if err == nil || errors.Is(err, io.EOF) {
538+ return nil
539+ }
540+ return err
541+}
A internal/app/broker_offline_mux_test.go +709 -0
@@ -0,0 +1,709 @@
1+package app
2+
3+import (
4+ "bytes"
5+ "context"
6+ "encoding/base64"
7+ "encoding/json"
8+ "errors"
9+ "io"
10+ "log/slog"
11+ "os"
12+ "path/filepath"
13+ "strconv"
14+ "strings"
15+ "syscall"
16+ "testing"
17+ "time"
18+ "unicode"
19+
20+ "github.com/stretchr/testify/require"
21+
22+ "github.com/bnema/vev/internal/adapters/brokerconfig"
23+ "github.com/bnema/vev/internal/adapters/brokeripc"
24+ "github.com/bnema/vev/internal/adapters/quic"
25+ "github.com/bnema/vev/internal/ports"
26+ "github.com/bnema/vev/internal/protocol"
27+)
28+
29+// Real-composition coverage for the P3.4 slice E remote mux helpers.
30+//
31+// The SSH carriage is exercised with an owned test subprocess: a fake `ssh`
32+// on PATH re-executes this test binary through TestMain's broker-helper
33+// interception, so the real helper code runs with real pipes and a real unix
34+// daemonmux fixture, and the exact ssh argv is captured for inspection. The
35+// QUIC carriage runs a real loopback authenticated QUIC server, so a successful
36+// round trip proves pin, token, nonce, and daemonmux handshake all compose.
37+
38+const (
39+ fakeSSHArgsEnv = "VEV_TEST_SSH_ARGS"
40+ fakeSSHReadyEnv = "VEV_TEST_SSH_READY"
41+ fakeSSHPIDEnv = "VEV_TEST_SSH_PID"
42+ fakeSSHBinEnv = "VEV_TEST_SSH_BIN"
43+ fakeSSHModeEnv = "VEV_TEST_SSH_MODE"
44+ fakeSSHRootEnv = "VEV_TEST_SSH_REMOTE_ROOT"
45+)
46+
47+// fakeSSHExecHelper is an owned test ssh: it records its exact argv and then
48+// re-executes this test binary as the named hidden mux helper against an
49+// isolated remote sandbox. Its stdin/stdout remain the carriage pipes.
50+const fakeSSHExecHelper = `#!/bin/sh
51+printf '%s\n' "$@" > "$` + fakeSSHArgsEnv + `"
52+exec "$` + fakeSSHBinEnv + `" "$` + fakeSSHModeEnv + `" --offline-root "$` + fakeSSHRootEnv + `"
53+`
54+
55+// fakeSSHEmitReadiness prints one controlled readiness line and exits, so a
56+// test can drive the broker-side pinned QUIC dial without a real remote host.
57+const fakeSSHEmitReadiness = `#!/bin/sh
58+printf '%s\n' "$@" > "$` + fakeSSHArgsEnv + `"
59+printf '%s\n' "$` + fakeSSHReadyEnv + `"
60+exit 0
61+`
62+
63+// fakeSSHSleep blocks without producing readiness, so a test can prove setup
64+// cancellation reaps the bootstrap child.
65+const fakeSSHSleep = `#!/bin/sh
66+printf '%s\n' "$@" > "$` + fakeSSHArgsEnv + `"
67+printf '%s\n' "$$" > "$` + fakeSSHPIDEnv + `"
68+sleep 30
69+`
70+
71+// fakeSSHFloodStderr floods stderr with text carrying a terminal escape and a
72+// control byte and never prints readiness, so a test can prove the broker-side
73+// diagnostic sink is read only after the os/exec copy goroutine has been killed
74+// and joined.
75+const fakeSSHFloodStderr = `#!/bin/sh
76+printf '%s\n' "$@" > "$` + fakeSSHArgsEnv + `"
77+printf '%s\n' "$$" > "$` + fakeSSHPIDEnv + `"
78+i=0
79+while true; do
80+ printf 'flood-%s\033[31mred\033[0m\001\002-noise\n' "$i" >&2
81+ i=$((i+1))
82+ sleep 0.01
83+done
84+`
85+
86+// fakeSSHStallNeverReady stalls without printing readiness and without exiting,
87+// so a test can prove the independent broker-side setup timeout interrupts a
88+// bootstrap that the caller never cancels. It execs the blocking sleep so the
89+// reaped process owns the captured stderr pipe and no grandchild lingers.
90+const fakeSSHStallNeverReady = `#!/bin/sh
91+printf '%s\n' "$@" > "$` + fakeSSHArgsEnv + `"
92+printf '%s\n' "$$" > "$` + fakeSSHPIDEnv + `"
93+exec sleep 30
94+`
95+
96+// installFakeSSH puts an owned `ssh` executable first on PATH and returns the
97+// file its argv is recorded to.
98+func installFakeSSH(t *testing.T, script string) string {
99+ t.Helper()
100+ binDir := t.TempDir()
101+ require.NoError(t, os.Chmod(binDir, 0o755))
102+ argsFile := filepath.Join(t.TempDir(), "ssh-args")
103+ require.NoError(t, os.WriteFile(filepath.Join(binDir, "ssh"), []byte(script), 0o755))
104+ t.Setenv("PATH", binDir+string(os.PathListSeparator)+os.Getenv("PATH"))
105+ t.Setenv(fakeSSHArgsEnv, argsFile)
106+ t.Setenv(fakeSSHBinEnv, os.Args[0])
107+ return argsFile
108+}
109+
110+// muxTestEnv isolates production XDG roots and records every detached mux
111+// helper so no test leaks a process.
112+type muxTestEnv struct {
113+ t *testing.T
114+ prodRuntime, prodState string
115+ recordDir string
116+}
117+
118+func newMuxTestEnv(t *testing.T) *muxTestEnv {
119+ t.Helper()
120+ _, prodRuntime, prodState := isolateSandboxEnv(t)
121+ recordDir := filepath.Join(shortTempDir(t, "vevm"), "r")
122+ require.NoError(t, os.MkdirAll(recordDir, 0o700))
123+ t.Setenv(brokerHelperEnv, "1")
124+ t.Setenv(brokerHelperRecordDirEnv, recordDir)
125+ env := &muxTestEnv{t: t, prodRuntime: prodRuntime, prodState: prodState, recordDir: recordDir}
126+ t.Cleanup(env.cleanup)
127+ return env
128+}
129+
130+// cleanup kills every recorded mux helper process.
131+func (e *muxTestEnv) cleanup() {
132+ for _, role := range []string{brokerMuxStdioCommand, brokerMuxQUICBootstrapCommand, brokerMuxQUICProxyCommand} {
133+ raw, err := os.ReadFile(filepath.Join(e.recordDir, role+".pids"))
134+ if err != nil {
135+ continue
136+ }
137+ for _, field := range strings.Fields(string(raw)) {
138+ pid, convErr := strconv.Atoi(field)
139+ if convErr != nil {
140+ continue
141+ }
142+ if err := syscall.Kill(pid, syscall.SIGKILL); err != nil && !errors.Is(err, syscall.ESRCH) {
143+ continue
144+ }
145+ _ = waitForProcessExit(pid, 5*time.Second)
146+ }
147+ }
148+}
149+
150+// writeMuxRoot writes one marker-valid sandbox root carrying a single
151+// registration with the supplied route object.
152+func writeMuxRoot(t *testing.T, route any) string {
153+ t.Helper()
154+ root := filepath.Join(shortTempDir(t, "vevx"), "s")
155+ require.NoError(t, os.MkdirAll(root, 0o700))
156+ writeSandboxConfig(t, root, sandboxRegistrationDocument(route))
157+ return root
158+}
159+
160+// loadTestRoute loads one sandbox configuration and returns the route its
161+// resolver publishes, so a test can inspect a parsed route without the
162+// unexported brokerconfig fields.
163+func loadTestRoute(t *testing.T, route any) brokerconfig.Route {
164+ t.Helper()
165+ root := writeMuxRoot(t, route)
166+ layout, err := offlineLayout(root)
167+ require.NoError(t, err)
168+ config, err := brokerconfig.Load(layout)
169+ require.NoError(t, err)
170+ resolved, err := config.Resolver().Resolve(context.Background(), ports.BrokerOpenStreamRequest{
171+ Purpose: ports.BrokerStreamControl,
172+ Endpoint: brokerTestEndpoint,
173+ Registration: brokerTestRegistration(),
174+ Policy: brokerTestPolicy(),
175+ })
176+ require.NoError(t, err)
177+ parsed, ok := config.RouteByAddress(resolved.Address)
178+ require.True(t, ok)
179+ return parsed
180+}
181+
182+// TestParseBrokerMuxArgs pins the strict hidden-helper argument contract.
183+func TestParseBrokerMuxArgs(t *testing.T) {
184+ tests := []struct {
185+ name string
186+ args []string
187+ wantErr string
188+ want brokerMuxOptions
189+ }{
190+ {name: "valid", args: []string{"--offline-root", "/srv/sandbox"}, want: brokerMuxOptions{offlineRoot: "/srv/sandbox"}},
191+ {name: "missing root", args: nil, wantErr: "requires --offline-root"},
192+ {name: "empty root", args: []string{"--offline-root", ""}, wantErr: "requires a path"},
193+ {name: "duplicate root", args: []string{"--offline-root", "/a", "--offline-root", "/b"}, wantErr: "duplicate"},
194+ {name: "unknown flag", args: []string{"--idle-grace", "1m"}, wantErr: "unknown flag"},
195+ {name: "positional", args: []string{"--offline-root", "/a", "extra"}, wantErr: "positional"},
196+ }
197+ for _, tt := range tests {
198+ t.Run(tt.name, func(t *testing.T) {
199+ command, err := parseBrokerMuxArgs(brokerMuxStdioCommand, kindBrokerMuxStdio, tt.args)
200+ if tt.wantErr != "" {
201+ require.Error(t, err)
202+ require.Contains(t, err.Error(), tt.wantErr)
203+ return
204+ }
205+ require.NoError(t, err)
206+ require.Equal(t, brokerMuxOptions{offlineRoot: tt.want.offlineRoot}, command.brokerMux)
207+ })
208+ }
209+}
210+
211+// TestBrokerMuxHelpersHiddenFromPublicHelp proves the three helpers parse but
212+// stay out of public help and never alter ordinary commands.
213+func TestBrokerMuxHelpersHiddenFromPublicHelp(t *testing.T) {
214+ for _, tt := range []struct {
215+ name string
216+ kind cmdKind
217+ }{
218+ {brokerMuxStdioCommand, kindBrokerMuxStdio},
219+ {brokerMuxQUICBootstrapCommand, kindBrokerMuxQUICBootstrap},
220+ {brokerMuxQUICProxyCommand, kindBrokerMuxQUICProxy},
221+ } {
222+ require.NotContains(t, usageText, tt.name)
223+ command, err := parseArgs([]string{tt.name, "--offline-root", "/srv/sandbox"})
224+ require.NoError(t, err)
225+ require.Equal(t, tt.kind, command.kind)
226+ require.Equal(t, "/srv/sandbox", command.brokerMux.offlineRoot)
227+ }
228+ // Ordinary commands are untouched by the new hidden entries.
229+ attach, err := parseArgs(nil)
230+ require.NoError(t, err)
231+ require.Equal(t, kindAttach, attach.kind)
232+ list, err := parseArgs([]string{"ls"})
233+ require.NoError(t, err)
234+ require.Equal(t, kindList, list.kind)
235+}
236+
237+// TestSSHMuxCommandSpecPreservesTrustAndNoPTY proves the ssh argv keeps `-T`,
238+// the option terminator, and the quoted remote argv, splices only the explicit
239+// trust inputs, and never disables host-key verification.
240+func TestSSHMuxCommandSpecPreservesTrustAndNoPTY(t *testing.T) {
241+ isolateSandboxEnv(t)
242+ route := loadTestRoute(t, map[string]any{
243+ "kind": "ssh-stdio",
244+ "target": "user@host:2222",
245+ "argv": []any{"vev", brokerMuxStdioCommand, "--offline-root", "/srv/remote"},
246+ "trust": map[string]any{"knownHostsFile": "/etc/ssh/known_hosts", "connectTimeout": "30s"},
247+ })
248+ spec := sshMuxCommandSpec(route)
249+ require.Equal(t, "ssh", spec.Path)
250+ require.Equal(t, "-T", spec.Args[0])
251+ require.Contains(t, spec.Args, "--")
252+ require.Contains(t, spec.Args, "UserKnownHostsFile=/etc/ssh/known_hosts")
253+ require.Contains(t, spec.Args, "ConnectTimeout=30")
254+ joined := strings.Join(spec.Args, " ")
255+ require.Contains(t, joined, "-- user@host:2222 ")
256+ require.Contains(t, joined, brokerMuxStdioCommand)
257+ require.NotContains(t, joined, "StrictHostKeyChecking")
258+ require.NotContains(t, joined, "BatchMode")
259+ require.NotContains(t, joined, "ProxyCommand")
260+ // The option terminator protects the target, and the -T option comes first.
261+ require.Less(t, indexOf(spec.Args, "-T"), indexOf(spec.Args, "--"))
262+ for _, arg := range spec.Args {
263+ require.NotEqual(t, "no", strings.ToLower(arg))
264+ }
265+
266+ // A route without trust inputs reuses the mux command construction
267+ // unchanged (`-T -- target remote`).
268+ plain := loadTestRoute(t, map[string]any{
269+ "kind": "ssh-quic",
270+ "target": "host",
271+ "argv": []any{"vev", brokerMuxQUICBootstrapCommand, "--offline-root", "/srv/remote"},
272+ })
273+ plainSpec := sshMuxCommandSpec(plain)
274+ require.Equal(t, []string{"-T", "--"}, append([]string(nil), plainSpec.Args[:2]...))
275+ require.NotContains(t, plainSpec.Args, "-o")
276+}
277+
278+// TestSSHMuxCommandSpecTargetQuotesSpacesInsideArgv proves a remote argv word
279+// with a space stays one quoted word and never becomes a second option.
280+func TestSSHMuxCommandSpecTargetQuotesSpacesInsideArgv(t *testing.T) {
281+ isolateSandboxEnv(t)
282+ route := loadTestRoute(t, map[string]any{
283+ "kind": "ssh-stdio",
284+ "target": "host",
285+ "argv": []any{"vev", "_broker-mux-stdio", "--offline-root", "/srv/with space"},
286+ })
287+ spec := sshMuxCommandSpec(route)
288+ require.Contains(t, spec.Args[len(spec.Args)-1], `'/srv/with space'`)
289+}
290+
291+// TestBrokerQUICPeerAddr pins the host-independent address composition.
292+func TestBrokerQUICPeerAddr(t *testing.T) {
293+ tests := []struct {
294+ target string
295+ port int
296+ want string
297+ wantErr bool
298+ }{
299+ {target: "user@host", port: 2222, want: "host:2222"},
300+ {target: "user@host:22", port: 2222, want: "host:2222"},
301+ {target: "host", port: 1, want: "host:1"},
302+ {target: "user@[::1]", port: 7, want: "[::1]:7"},
303+ {target: "host", port: 0, wantErr: true},
304+ {target: "host", port: 65536, wantErr: true},
305+ {target: "user@", port: 7, wantErr: true},
306+ }
307+ for _, tt := range tests {
308+ got, err := brokerQUICPeerAddr(tt.target, tt.port)
309+ if tt.wantErr {
310+ require.Error(t, err)
311+ continue
312+ }
313+ require.NoError(t, err)
314+ require.Equal(t, tt.want, got)
315+ }
316+}
317+
318+// TestBrokerMuxConnectorRejectsUnknownAddress proves the transport selector
319+// refuses any address the immutable configuration did not produce.
320+func TestBrokerMuxConnectorRejectsUnknownAddress(t *testing.T) {
321+ isolateSandboxEnv(t)
322+ root := writeMuxRoot(t, map[string]any{"kind": "unix", "path": "/tmp/unused-mux.sock"})
323+ layout, err := offlineLayout(root)
324+ require.NoError(t, err)
325+ config, err := brokerconfig.Load(layout)
326+ require.NoError(t, err)
327+ connector, err := brokerMuxConnector(config, slog.New(slog.NewTextHandler(io.Discard, nil)))
328+ require.NoError(t, err)
329+
330+ _, err = connector.Connect(context.Background(), ports.BrokerResolvedEndpoint{
331+ Identity: brokerTestIdentity,
332+ Policy: brokerTestPolicy(),
333+ Address: "offline-route-00000000000000000000000000000000",
334+ })
335+ require.Error(t, err)
336+ var typed ports.BrokerError
337+ require.ErrorAs(t, err, &typed)
338+ require.Equal(t, ports.BrokerErrorUnavailable, typed.Code)
339+}
340+
341+// TestBrokerServeSSHStdioRouteRoundTrip drives the full broker path over an SSH
342+// stdio route: the broker dials the fake ssh child, the hidden helper bridges
343+// its stdio to a real unix daemonmux fixture, and a typed Ping returns a Pong.
344+func TestBrokerServeSSHStdioRouteRoundTrip(t *testing.T) {
345+ env := newMuxTestEnv(t)
346+ policy := brokerTestPolicy()
347+ fixture := newBrokerMuxFixture(t, policy)
348+ remoteRoot := writeMuxRoot(t, map[string]any{"kind": "unix", "path": fixture.route})
349+ argsFile := installFakeSSH(t, fakeSSHExecHelper)
350+ t.Setenv(fakeSSHModeEnv, brokerMuxStdioCommand)
351+ t.Setenv(fakeSSHRootEnv, remoteRoot)
352+ localRoot := writeMuxRoot(t, map[string]any{
353+ "kind": "ssh-stdio",
354+ "target": "user@muxhost:2222",
355+ "argv": []any{"vev", brokerMuxStdioCommand, "--offline-root", remoteRoot},
356+ "trust": map[string]any{"knownHostsFile": "/etc/ssh/known_hosts", "connectTimeout": "5s"},
357+ })
358+
359+ socketPath, stop := startMuxBrokerServe(t, localRoot)
360+ defer stop()
361+ requireBrokerPing(t, socketPath)
362+ require.Positive(t, fixture.streams.Load())
363+
364+ // The exact ssh argv was built by BuildCommandForMux plus the trust inputs.
365+ argv := readFileString(t, argsFile)
366+ require.Contains(t, argv, "-T\n")
367+ require.Contains(t, argv, "--\nuser@muxhost:2222\n")
368+ require.Contains(t, argv, brokerMuxStdioCommand)
369+ require.Contains(t, argv, "UserKnownHostsFile=/etc/ssh/known_hosts")
370+ require.Contains(t, argv, "ConnectTimeout=5")
371+ require.NotContains(t, argv, "StrictHostKeyChecking")
372+ require.NotContains(t, argv, "BatchMode")
373+
374+ stop()
375+ env.requireProductionUntouched(t)
376+}
377+
378+// TestBrokerServeSSHQUICRouteRoundTrip drives the full broker path over an
379+// SSH-authenticated QUIC route with a real loopback QUIC server and a real
380+// one-time credential.
381+func TestBrokerServeSSHQUICRouteRoundTrip(t *testing.T) {
382+ env := newMuxTestEnv(t)
383+ policy := brokerTestPolicy()
384+ fixture := newBrokerMuxFixture(t, policy)
385+ remoteRoot := writeMuxRoot(t, map[string]any{"kind": "unix", "path": fixture.route})
386+ installFakeSSH(t, fakeSSHExecHelper)
387+ t.Setenv(fakeSSHModeEnv, brokerMuxQUICBootstrapCommand)
388+ t.Setenv(fakeSSHRootEnv, remoteRoot)
389+ localRoot := writeMuxRoot(t, map[string]any{
390+ "kind": "ssh-quic",
391+ "target": "127.0.0.1",
392+ "argv": []any{"vev", brokerMuxQUICBootstrapCommand, "--offline-root", remoteRoot},
393+ })
394+
395+ socketPath, stop := startMuxBrokerServe(t, localRoot)
396+ defer stop()
397+ requireBrokerPing(t, socketPath)
398+ require.Positive(t, fixture.streams.Load())
399+
400+ stop()
401+ env.requireProductionUntouched(t)
402+}
403+
404+// TestBrokerMuxQUICRejectsBadPinWithoutLeakingSecret proves the pinned dial
405+// fails closed on a wrong certificate pin and that the one-time credential
406+// never appears in the returned error.
407+func TestBrokerMuxQUICRejectsBadPinWithoutLeakingSecret(t *testing.T) {
408+ isolateSandboxEnv(t)
409+ installFakeSSH(t, fakeSSHEmitReadiness)
410+
411+ server, readiness, err := quic.NewServer()
412+ require.NoError(t, err)
413+ defer func() { _ = server.Close() }()
414+ // Flip the pin: the TLS handshake must fail before any token is accepted.
415+ tampered := readiness
416+ tampered.Fingerprint = strings.Repeat("00", 32)
417+ t.Setenv(fakeSSHReadyEnv, string(mustJSON(t, tampered)))
418+
419+ route := loadTestRoute(t, map[string]any{
420+ "kind": "ssh-quic",
421+ "target": "127.0.0.1",
422+ "argv": []any{"vev", brokerMuxQUICBootstrapCommand, "--offline-root", "/srv/remote"},
423+ })
424+ ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
425+ defer cancel()
426+ _, err = dialBrokerRoute(ctx, route, slog.New(slog.NewTextHandler(io.Discard, nil)))
427+ require.Error(t, err)
428+ requireBrokerErrorHasNoReadiness(t, err, readiness)
429+}
430+
431+// TestBrokerMuxQUICRejectsExpiredReadiness proves client-side parsing refuses
432+// an implausibly old credential before dialing and never echoes its secret.
433+func TestBrokerMuxQUICRejectsExpiredReadiness(t *testing.T) {
434+ isolateSandboxEnv(t)
435+ installFakeSSH(t, fakeSSHEmitReadiness)
436+
437+ token := base64.StdEncoding.EncodeToString([]byte(strings.Repeat("A", 32)))
438+ nonce := base64.StdEncoding.EncodeToString([]byte(strings.Repeat("B", 16)))
439+ readiness := quic.Readiness{
440+ Version: 1,
441+ Port: 2222,
442+ Token: token,
443+ Nonce: nonce,
444+ Fingerprint: strings.Repeat("00", 32),
445+ ExpiresAt: time.Now().Add(-time.Hour).Unix(),
446+ }
447+ t.Setenv(fakeSSHReadyEnv, string(mustJSON(t, readiness)))
448+
449+ route := loadTestRoute(t, map[string]any{
450+ "kind": "ssh-quic",
451+ "target": "127.0.0.1",
452+ "argv": []any{"vev", brokerMuxQUICBootstrapCommand, "--offline-root", "/srv/remote"},
453+ })
454+ _, err := dialBrokerRoute(context.Background(), route, slog.New(slog.NewTextHandler(io.Discard, nil)))
455+ require.Error(t, err)
456+ require.NotContains(t, err.Error(), token)
457+ require.NotContains(t, err.Error(), nonce)
458+}
459+
460+// TestBrokerMuxDialCancellationReapsBootstrap proves one setup context bounds
461+// the bootstrap wait, a cancellation returns promptly, and the bootstrap child
462+// is killed and reaped rather than leaked.
463+func TestBrokerMuxDialCancellationReapsBootstrap(t *testing.T) {
464+ isolateSandboxEnv(t)
465+ installFakeSSH(t, fakeSSHSleep)
466+ pidFile := filepath.Join(t.TempDir(), "ssh.pid")
467+ t.Setenv(fakeSSHPIDEnv, pidFile)
468+
469+ route := loadTestRoute(t, map[string]any{
470+ "kind": "ssh-quic",
471+ "target": "127.0.0.1",
472+ "argv": []any{"vev", brokerMuxQUICBootstrapCommand, "--offline-root", "/srv/remote"},
473+ })
474+ ctx, cancel := context.WithTimeout(context.Background(), 150*time.Millisecond)
475+ defer cancel()
476+ start := time.Now()
477+ _, err := dialBrokerRoute(ctx, route, slog.New(slog.NewTextHandler(io.Discard, nil)))
478+ require.Error(t, err)
479+ require.Less(t, time.Since(start), 3*time.Second, "cancellation must return promptly")
480+
481+ pid := readPID(t, pidFile)
482+ require.NoError(t, waitForProcessExit(pid, 3*time.Second), "the bootstrap child must be reaped")
483+}
484+
485+// TestBrokerMuxDialFloodedStderrIsReapedAndSanitized is the regression for the
486+// stderr DiagnosticSink race: a bootstrap that floods stderr and never prints
487+// readiness must be killed and reaped before the captured sink is read, and the
488+// diagnostic that reaches the log must stay bounded and control-byte-free. Run
489+// under -race with -count>1, this fails if the os/exec copy goroutine is ever
490+// observed while it is still writing the sink.
491+func TestBrokerMuxDialFloodedStderrIsReapedAndSanitized(t *testing.T) {
492+ isolateSandboxEnv(t)
493+ installFakeSSH(t, fakeSSHFloodStderr)
494+ pidFile := filepath.Join(t.TempDir(), "ssh.pid")
495+ t.Setenv(fakeSSHPIDEnv, pidFile)
496+
497+ route := loadTestRoute(t, map[string]any{
498+ "kind": "ssh-quic",
499+ "target": "127.0.0.1",
500+ "argv": []any{"vev", brokerMuxQUICBootstrapCommand, "--offline-root", "/srv/remote"},
501+ })
502+
503+ var logs bytes.Buffer
504+ logger := slog.New(slog.NewJSONHandler(&logs, nil))
505+ ctx, cancel := context.WithTimeout(context.Background(), 400*time.Millisecond)
506+ defer cancel()
507+ _, err := dialBrokerRoute(ctx, route, logger)
508+ require.Error(t, err)
509+ require.NotContains(t, err.Error(), "flood", "bootstrap stderr must never reach the error")
510+
511+ // Reaping is observable: the flooding bootstrap is gone, not leaked.
512+ pid := readPID(t, pidFile)
513+ require.NoError(t, waitForProcessExit(pid, 3*time.Second), "the flooding bootstrap must be reaped")
514+
515+ record := decodeBootstrapDiagnostic(t, logs.Bytes())
516+ require.NotEmpty(t, record.Diagnostic)
517+ require.LessOrEqual(t, len(record.Diagnostic), 512, "the captured diagnostic must stay bounded")
518+ for _, r := range record.Diagnostic {
519+ require.False(t, unicode.IsControl(r) && r != '\t' && r != '\n', "control byte %q leaked into the diagnostic", r)
520+ }
521+}
522+
523+// TestBrokerMuxDialStalledBootstrapUsesSetupTimeout proves the independent
524+// broker-side setup timeout bounds the readiness wait: with no caller deadline a
525+// helper that stalls without printing readiness is still interrupted within the
526+// bound, and its child is killed and reaped. It is deterministic because the
527+// effective bound is narrowed to a test value instead of the production one.
528+func TestBrokerMuxDialStalledBootstrapUsesSetupTimeout(t *testing.T) {
529+ isolateSandboxEnv(t)
530+ installFakeSSH(t, fakeSSHStallNeverReady)
531+ pidFile := filepath.Join(t.TempDir(), "ssh.pid")
532+ t.Setenv(fakeSSHPIDEnv, pidFile)
533+
534+ restore := brokerMuxBootstrapReadyTimeout
535+ brokerMuxBootstrapReadyTimeout = 250 * time.Millisecond
536+ t.Cleanup(func() { brokerMuxBootstrapReadyTimeout = restore })
537+
538+ route := loadTestRoute(t, map[string]any{
539+ "kind": "ssh-quic",
540+ "target": "127.0.0.1",
541+ "argv": []any{"vev", brokerMuxQUICBootstrapCommand, "--offline-root", "/srv/remote"},
542+ })
543+ start := time.Now()
544+ _, err := dialBrokerRoute(context.Background(), route, slog.New(slog.NewTextHandler(io.Discard, nil)))
545+ elapsed := time.Since(start)
546+ require.Error(t, err)
547+ require.Contains(t, err.Error(), "read bootstrap readiness")
548+ require.GreaterOrEqual(t, elapsed, 200*time.Millisecond, "the independent bound must not return early")
549+ require.Less(t, elapsed, 5*time.Second, "the independent bound must interrupt a stalled bootstrap")
550+
551+ pid := readPID(t, pidFile)
552+ require.NoError(t, waitForProcessExit(pid, 3*time.Second), "the stalled bootstrap must be reaped")
553+}
554+
555+// TestBrokerServeUnixRouteRoundTrip proves the transport selector also handles
556+// the explicit unix route variant, not just the legacy path string.
557+func TestBrokerServeUnixRouteRoundTrip(t *testing.T) {
558+ env := newMuxTestEnv(t)
559+ fixture := newBrokerMuxFixture(t, brokerTestPolicy())
560+ localRoot := writeMuxRoot(t, map[string]any{"kind": "unix", "path": fixture.route})
561+
562+ socketPath, stop := startMuxBrokerServe(t, localRoot)
563+ defer stop()
564+ requireBrokerPing(t, socketPath)
565+ require.Positive(t, fixture.streams.Load())
566+
567+ stop()
568+ env.requireProductionUntouched(t)
569+}
570+
571+// startMuxBrokerServe starts one offline broker over localRoot and returns its
572+// socket path plus an idempotent stop.
573+func startMuxBrokerServe(t *testing.T, localRoot string) (string, func()) {
574+ t.Helper()
575+ deps, ready := testBrokerServeDeps(newSandboxClock())
576+ ctx, cancel := context.WithCancel(context.Background())
577+ done := runSandbox(ctx, brokerServeOptions{offlineRoot: localRoot}, deps)
578+ socketPath := awaitSandboxReady(t, ready)
579+ var once bool
580+ stop := func() {
581+ if once {
582+ return
583+ }
584+ once = true
585+ awaitSandboxCancel(t, cancel, done)
586+ requireSocketRemoved(t, socketPath)
587+ }
588+ return socketPath, stop
589+}
590+
591+// requireBrokerPing opens one control stream and requires one Pong.
592+func requireBrokerPing(t *testing.T, socketPath string) {
593+ t.Helper()
594+ clientCtx, clientCancel := context.WithTimeout(context.Background(), brokerTestWait)
595+ defer clientCancel()
596+ service, err := brokeripc.Dial(clientCtx, socketPath, brokeripc.Config{})
597+ require.NoError(t, err)
598+ defer func() { _ = service.Close() }()
599+
600+ stream, err := service.OpenStream(clientCtx, ports.BrokerOpenStreamRequest{
601+ Purpose: ports.BrokerStreamControl,
602+ Endpoint: brokerTestEndpoint,
603+ Registration: brokerTestRegistration(),
604+ Policy: brokerTestPolicy(),
605+ })
606+ require.NoError(t, err)
607+ defer func() { _ = stream.Close() }()
608+
609+ type pingResult struct {
610+ message protocol.ServerMessage
611+ err error
612+ }
613+ result := make(chan pingResult, 1)
614+ go func() {
615+ if sendErr := stream.SendClient(protocol.Ping{}); sendErr != nil {
616+ result <- pingResult{err: sendErr}
617+ return
618+ }
619+ message, recvErr := stream.ReceiveServer()
620+ result <- pingResult{message: message, err: recvErr}
621+ }()
622+ select {
623+ case outcome := <-result:
624+ require.NoError(t, outcome.err)
625+ require.Equal(t, protocol.Pong{}, outcome.message)
626+ case <-time.After(brokerTestWait):
627+ t.Fatal("typed broker -> mux round trip timed out")
628+ }
629+}
630+
631+// requireBrokerErrorHasNoReadiness asserts no readiness secret or stderr leaks
632+// into a returned error.
633+func requireBrokerErrorHasNoReadiness(t *testing.T, err error, readiness quic.Readiness) {
634+ t.Helper()
635+ message := err.Error()
636+ require.NotContains(t, message, readiness.Token)
637+ require.NotContains(t, message, readiness.Nonce)
638+ require.NotContains(t, message, readiness.Fingerprint)
639+ require.NotContains(t, message, "token")
640+ require.NotContains(t, message, "nonce")
641+}
642+
643+// requireProductionUntouched asserts no production runtime or state entry was
644+// created by the composition or its helpers.
645+func (e *muxTestEnv) requireProductionUntouched(t *testing.T) {
646+ t.Helper()
647+ requireProductionUntouched(t, e.prodRuntime, e.prodState)
648+ entries, err := os.ReadDir(e.prodRuntime)
649+ require.NoError(t, err)
650+ for _, entry := range entries {
651+ require.NotContains(t, entry.Name(), "daemon.sock", "helpers must never start an ordinary daemon")
652+ }
653+}
654+
655+func mustJSON(t *testing.T, value any) []byte {
656+ t.Helper()
657+ raw, err := json.Marshal(value)
658+ require.NoError(t, err)
659+ return raw
660+}
661+
662+func readFileString(t *testing.T, path string) string {
663+ t.Helper()
664+ raw, err := os.ReadFile(path)
665+ require.NoError(t, err)
666+ return string(raw)
667+}
668+
669+func readPID(t *testing.T, path string) int {
670+ t.Helper()
671+ deadline := time.Now().Add(3 * time.Second)
672+ for {
673+ raw, err := os.ReadFile(path)
674+ if err == nil {
675+ pid, convErr := strconv.Atoi(strings.TrimSpace(string(raw)))
676+ require.NoError(t, convErr)
677+ return pid
678+ }
679+ if !time.Now().Before(deadline) {
680+ t.Fatalf("read pid file %s: %v", path, err)
681+ }
682+ time.Sleep(10 * time.Millisecond)
683+ }
684+}
685+
686+func indexOf(values []string, want string) int {
687+ for i, value := range values {
688+ if value == want {
689+ return i
690+ }
691+ }
692+ return -1
693+}
694+
695+// bootstrapDiagnostic is one decoded broker_mux_bootstrap_stderr log record.
696+type bootstrapDiagnostic struct {
697+ Message string `json:"msg"`
698+ Diagnostic string `json:"diagnostic"`
699+}
700+
701+// decodeBootstrapDiagnostic decodes the single sanitized diagnostic the broker
702+// logs for a failed bootstrap.
703+func decodeBootstrapDiagnostic(t *testing.T, raw []byte) bootstrapDiagnostic {
704+ t.Helper()
705+ var record bootstrapDiagnostic
706+ require.NoError(t, json.Unmarshal(bytes.TrimSpace(raw), &record))
707+ require.Equal(t, "broker_mux_bootstrap_stderr", record.Message)
708+ return record
709+}
A internal/app/broker_offline_status.go +674 -0
@@ -0,0 +1,674 @@
1+package app
2+
3+import (
4+ "bytes"
5+ "context"
6+ "encoding/json"
7+ "errors"
8+ "fmt"
9+ "io"
10+ "io/fs"
11+ "net"
12+ "os"
13+ "os/exec"
14+ "os/signal"
15+ "strings"
16+ "syscall"
17+ "time"
18+
19+ "github.com/bnema/vev/internal/adapters/brokerconfig"
20+ "github.com/bnema/vev/internal/adapters/brokeripc"
21+ "github.com/bnema/vev/internal/adapters/lifecycle"
22+ "github.com/bnema/vev/internal/platform"
23+ "github.com/bnema/vev/internal/ports"
24+ "github.com/bnema/vev/pkg/safedir"
25+)
26+
27+// Detached connect-or-spawn and status (Plan 001 P3.4 slice D).
28+//
29+// `_broker-launcher` is the intermediate half of a double fork: it starts one
30+// hidden `_broker-serve` in a new session and exits immediately, exactly like
31+// the daemon launcher, so the broker is reparented before the status caller
32+// continues. `_broker-status` is the client-facing probe: without --ensure it
33+// is dial-only and renders the observable state, and with --ensure it
34+// coordinates a race-safe connect-or-spawn.
35+//
36+// Both commands are hidden from public help and change no ordinary command,
37+// path, or factory. They operate only inside the operator-supplied offline
38+// root; a root that overlaps the production runtime or state directories is
39+// refused before anything is created.
40+//
41+// Race safety rests on two exclusive descriptor-backed locks that are never
42+// stolen by age and whose inodes are never unlinked:
43+//
44+// - the broker lifetime lock (`layout.Runtime/lifecycle.lock`) is held by the
45+// foreground `_broker-serve` for its whole run; a status probe uses it only
46+// to detect that a broker already owns the sandbox and therefore must not
47+// be duplicated.
48+// - the spawn-election lock (`layout.Spawn/lifecycle.lock`) elects exactly
49+// one status caller to spawn. Holding it across the readiness wait keeps
50+// late callers waiting on the winner instead of racing a second spawn; if
51+// the winner dies, the kernel releases the lock, so a waiter can take over
52+// without ever unlinking or age-stealing the lock file.
53+//
54+// The elected spawner holds the election until the endpoint answers a valid
55+// broker registration and publishes its initial snapshot, or until the one
56+// absolute overall deadline expires. A live-but-incompatible endpoint and a
57+// stalled handshake are diagnosed without spawning.
58+
59+const (
60+ // brokerLauncherCommand is the hidden intermediate that starts one detached
61+ // `_broker-serve` and exits.
62+ brokerLauncherCommand = "_broker-launcher"
63+ // brokerStatusCommand is the hidden broker status probe.
64+ brokerStatusCommand = "_broker-status"
65+)
66+
67+// Bounds for the detached connect-or-spawn path. Every wait is bounded by the
68+// one absolute overall deadline, and the first dial is bounded on its own so a
69+// stalled endpoint cannot consume the whole budget before classification.
70+const (
71+ brokerStatusDefaultTimeout = 10 * time.Second
72+ brokerStatusProbeBudget = 2 * time.Second
73+ brokerStatusInitialBackoff = 5 * time.Millisecond
74+ brokerStatusMaxBackoff = 50 * time.Millisecond
75+ brokerStatusMaxReportBytes = 4 << 10
76+)
77+
78+var (
79+ // errBrokerAbsent reports that no broker is bound at the endpoint, so a
80+ // spawn may recover it. A missing socket and a stale socket left by a dead
81+ // owner are both absence.
82+ errBrokerAbsent = errors.New("vev: broker endpoint is absent")
83+ // errBrokerIncompatible reports that the endpoint exists but is not a
84+ // compatible broker (a foreign path, a live endpoint that fails the broker
85+ // handshake, or a handshake that stalls past its bound). It is never an
86+ // invitation to spawn.
87+ errBrokerIncompatible = errors.New("vev: broker endpoint is not a compatible broker")
88+ // errBrokerNotReady reports that --ensure did not observe a ready broker
89+ // before the overall deadline.
90+ errBrokerNotReady = errors.New("vev: broker did not become ready")
91+)
92+
93+// brokerLauncherOptions is the parsed `_broker-launcher` invocation.
94+type brokerLauncherOptions struct {
95+ offlineRoot string
96+ idleGrace time.Duration
97+}
98+
99+// brokerStatusOptions is the parsed `_broker-status` invocation.
100+type brokerStatusOptions struct {
101+ offlineRoot string
102+ ensure bool
103+ timeout time.Duration
104+}
105+
106+// brokerStatusReport is the bounded JSON status document. It carries exactly
107+// the ready/offline state, the broker endpoint, and the broker's epoch,
108+// revision, and host count. Epoch, revision, and host count are zero when the
109+// broker is offline.
110+type brokerStatusReport struct {
111+ Status string `json:"status"`
112+ Endpoint string `json:"endpoint"`
113+ Epoch ports.BrokerEpoch `json:"epoch"`
114+ Revision ports.BrokerRevision `json:"revision"`
115+ HostCount int `json:"host_count"`
116+}
117+
118+// brokerStatusLock is one held exclusive election lock.
119+type brokerStatusLock interface {
120+ Release() error
121+}
122+
123+// brokerStatusDeps are the injectable seams of `_broker-status`. Every seam has
124+// a production default; tests substitute them to drive the election, deadline,
125+// and classification deterministically.
126+type brokerStatusDeps struct {
127+ now func() time.Time
128+ wait func(context.Context, time.Duration) error
129+ probe func(context.Context, string) (brokerStatusReport, error)
130+ acquireElection func(string) (brokerStatusLock, error)
131+ lifetimeOwner func(string) (bool, error)
132+ spawn func(ctx context.Context, root string, grace time.Duration, graceSet bool) error
133+ stdout io.Writer
134+}
135+
136+// defaultBrokerStatusDeps returns the production seams.
137+func defaultBrokerStatusDeps() brokerStatusDeps {
138+ return brokerStatusDeps{
139+ now: time.Now,
140+ wait: waitBackoff,
141+ probe: probeBrokerStatus,
142+ acquireElection: func(dir string) (brokerStatusLock, error) {
143+ return lifecycle.TryAcquire(dir)
144+ },
145+ lifetimeOwner: func(runtimeDir string) (bool, error) {
146+ owner, err := lifecycle.TryAcquire(runtimeDir)
147+ if err != nil {
148+ if errors.Is(err, lifecycle.ErrBusy) {
149+ return true, nil
150+ }
151+ return false, err
152+ }
153+ return false, owner.Release()
154+ },
155+ spawn: spawnBrokerLauncher,
156+ stdout: os.Stdout,
157+ }
158+}
159+
160+// parseBrokerLauncherArgs strictly parses
161+// `_broker-launcher --offline-root ABS [--idle-grace DURATION]`. Unknown flags,
162+// duplicate flags, positionals, and non-positive durations are refused.
163+func parseBrokerLauncherArgs(args []string) (command, error) {
164+ var options brokerLauncherOptions
165+ var seenRoot, seenGrace bool
166+ for index := 0; index < len(args); index++ {
167+ switch args[index] {
168+ case "--offline-root":
169+ if seenRoot {
170+ return command{}, usagef("`%s` received duplicate --offline-root", brokerLauncherCommand)
171+ }
172+ if index+1 >= len(args) || args[index+1] == "" {
173+ return command{}, usagef("`--offline-root` requires a path")
174+ }
175+ options.offlineRoot = args[index+1]
176+ seenRoot = true
177+ index++
178+ case "--idle-grace":
179+ if seenGrace {
180+ return command{}, usagef("`%s` received duplicate --idle-grace", brokerLauncherCommand)
181+ }
182+ if index+1 >= len(args) || args[index+1] == "" {
183+ return command{}, usagef("`--idle-grace` requires a duration")
184+ }
185+ grace, err := time.ParseDuration(args[index+1])
186+ if err != nil {
187+ return command{}, usagef("`--idle-grace` %q is not a duration", args[index+1])
188+ }
189+ if grace <= 0 {
190+ return command{}, usagef("`--idle-grace` must be positive")
191+ }
192+ options.idleGrace = grace
193+ seenGrace = true
194+ index++
195+ default:
196+ if strings.HasPrefix(args[index], "-") {
197+ return command{}, usagef("unknown flag %q for `%s`", args[index], brokerLauncherCommand)
198+ }
199+ return command{}, usagef("`%s` does not accept positional arguments", brokerLauncherCommand)
200+ }
201+ }
202+ if options.offlineRoot == "" {
203+ return command{}, usagef("`%s` requires --offline-root", brokerLauncherCommand)
204+ }
205+ return command{kind: kindBrokerLauncher, brokerLauncher: options}, nil
206+}
207+
208+// parseBrokerStatusArgs strictly parses
209+// `_broker-status --offline-root ABS [--ensure] [--timeout DURATION]`. Unknown
210+// flags, duplicate flags, positionals, and non-positive timeouts are refused.
211+func parseBrokerStatusArgs(args []string) (command, error) {
212+ options := brokerStatusOptions{timeout: brokerStatusDefaultTimeout}
213+ var seenRoot, seenEnsure, seenTimeout bool
214+ for index := 0; index < len(args); index++ {
215+ switch args[index] {
216+ case "--offline-root":
217+ if seenRoot {
218+ return command{}, usagef("`%s` received duplicate --offline-root", brokerStatusCommand)
219+ }
220+ if index+1 >= len(args) || args[index+1] == "" {
221+ return command{}, usagef("`--offline-root` requires a path")
222+ }
223+ options.offlineRoot = args[index+1]
224+ seenRoot = true
225+ index++
226+ case "--ensure":
227+ if seenEnsure {
228+ return command{}, usagef("`%s` received duplicate --ensure", brokerStatusCommand)
229+ }
230+ options.ensure = true
231+ seenEnsure = true
232+ case "--timeout":
233+ if seenTimeout {
234+ return command{}, usagef("`%s` received duplicate --timeout", brokerStatusCommand)
235+ }
236+ if index+1 >= len(args) || args[index+1] == "" {
237+ return command{}, usagef("`--timeout` requires a duration")
238+ }
239+ timeout, err := time.ParseDuration(args[index+1])
240+ if err != nil {
241+ return command{}, usagef("`--timeout` %q is not a duration", args[index+1])
242+ }
243+ if timeout <= 0 {
244+ return command{}, usagef("`--timeout` must be positive")
245+ }
246+ options.timeout = timeout
247+ seenTimeout = true
248+ index++
249+ default:
250+ if strings.HasPrefix(args[index], "-") {
251+ return command{}, usagef("unknown flag %q for `%s`", args[index], brokerStatusCommand)
252+ }
253+ return command{}, usagef("`%s` does not accept positional arguments", brokerStatusCommand)
254+ }
255+ }
256+ if options.offlineRoot == "" {
257+ return command{}, usagef("`%s` requires --offline-root", brokerStatusCommand)
258+ }
259+ return command{kind: kindBrokerStatus, brokerStatus: options}, nil
260+}
261+
262+// runBrokerLauncherCommand runs the hidden launcher. It changes no ordinary
263+// command and never runs in production flows.
264+func runBrokerLauncherCommand(_ context.Context, options brokerLauncherOptions) error {
265+ return runBrokerLauncher(options)
266+}
267+
268+// runBrokerLauncher validates the sandbox configuration, diagnoses an explicit
269+// idle grace that conflicts with the provisioned one, then starts one detached
270+// `_broker-serve` in a new session and returns.
271+func runBrokerLauncher(options brokerLauncherOptions) error {
272+ layout, err := offlineLayout(options.offlineRoot)
273+ if err != nil {
274+ return err
275+ }
276+ if err := safedir.EnsurePrivate(layout.Root); err != nil {
277+ return fmt.Errorf("vev: secure broker sandbox root: %w", err)
278+ }
279+ config, err := brokerconfig.Load(layout)
280+ if err != nil {
281+ return err
282+ }
283+ grace, err := effectiveIdleGrace(config, options.idleGrace)
284+ if err != nil {
285+ return err
286+ }
287+ args := []string{brokerServeCommand, "--offline-root", options.offlineRoot}
288+ if grace > 0 {
289+ args = append(args, "--idle-grace", grace.String())
290+ }
291+ return startDetachedBrokerServe(args)
292+}
293+
294+// startDetachedBrokerServe starts one `_broker-serve` in a new session and
295+// releases it, so the broker outlives the short-lived launcher. It mirrors the
296+// daemon launcher's double-fork boundary: no ambient variable is dropped beyond
297+// the performance-trace inputs, stdio is /dev/null, and the process is
298+// reparented immediately.
299+func startDetachedBrokerServe(args []string) error {
300+ exePath, err := selfExePath()
301+ if err != nil {
302+ return fmt.Errorf("vev: resolving executable path: %w", err)
303+ }
304+ devNull, err := os.OpenFile(os.DevNull, os.O_RDWR, 0)
305+ if err != nil {
306+ return fmt.Errorf("vev: opening %s: %w", os.DevNull, err)
307+ }
308+ defer func() { _ = devNull.Close() }()
309+
310+ cmd := exec.Command(exePath, args...)
311+ cmd.Env = withoutPerformanceTraceEnv(os.Environ())
312+ cmd.Dir = platform.DirOrHome("")
313+ cmd.Stdin = devNull
314+ cmd.Stdout = devNull
315+ cmd.Stderr = devNull
316+ cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
317+ if err := cmd.Start(); err != nil {
318+ return fmt.Errorf("vev: starting detached broker: %w", err)
319+ }
320+ return cmd.Process.Release()
321+}
322+
323+// spawnBrokerLauncher runs the hidden launcher, bounded by ctx, and waits for
324+// it to exit, so the broker has been reparented before the caller retries the
325+// endpoint. exec.CommandContext is deliberate: the overall status deadline or a
326+// signal kills a launcher that is still waiting, while killing the launcher
327+// after it has started the detached `_broker-serve` does not touch that serve --
328+// it is in its own session and was released, so it survives.
329+func spawnBrokerLauncher(ctx context.Context, root string, grace time.Duration, graceSet bool) error {
330+ exePath, err := selfExePath()
331+ if err != nil {
332+ return fmt.Errorf("vev: resolving executable path: %w", err)
333+ }
334+ devNull, err := os.OpenFile(os.DevNull, os.O_RDWR, 0)
335+ if err != nil {
336+ return fmt.Errorf("vev: opening %s: %w", os.DevNull, err)
337+ }
338+ defer func() { _ = devNull.Close() }()
339+
340+ args := []string{brokerLauncherCommand, "--offline-root", root}
341+ if graceSet {
342+ args = append(args, "--idle-grace", grace.String())
343+ }
344+ var stderr bytes.Buffer
345+ cmd := exec.CommandContext(ctx, exePath, args...)
346+ cmd.Env = withoutPerformanceTraceEnv(os.Environ())
347+ cmd.Dir = platform.DirOrHome("")
348+ cmd.Stdin = devNull
349+ cmd.Stdout = devNull
350+ cmd.Stderr = &stderr
351+ if err := cmd.Run(); err != nil {
352+ if detail := bytes.TrimSpace(stderr.Bytes()); len(detail) > 0 {
353+ return fmt.Errorf("vev: launching broker: %w: %s", err, detail)
354+ }
355+ return fmt.Errorf("vev: launching broker: %w", err)
356+ }
357+ return nil
358+}
359+
360+// runBrokerStatusCommand runs the hidden status probe, bounded by a signal-aware
361+// parent context.
362+func runBrokerStatusCommand(ctx context.Context, options brokerStatusOptions) error {
363+ ctx, stop := signal.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM)
364+ defer stop()
365+ return runBrokerStatus(ctx, options, defaultBrokerStatusDeps())
366+}
367+
368+// runBrokerStatus renders the broker status as one bounded JSON document.
369+//
370+// Without --ensure it is dial-only: it performs one probe bounded by
371+// --timeout and always succeeds, reporting ready or offline, so the JSON is the
372+// sole result and a caller never has to interpret an exit code to read a state.
373+// The timeout is the single absolute probe bound; it is never silently replaced
374+// by the ensure probe budget, so a caller can tighten it (or rely on the
375+// default) and observe the same bound. With --ensure it uses one absolute
376+// overall deadline; on success it prints the ready report and
377+// returns nil, and on any layout, configuration, or startup failure it still
378+// prints the offline report but returns a code-3 error so the caller always
379+// observes one offline JSON document and a distinct exit code.
380+func runBrokerStatus(ctx context.Context, options brokerStatusOptions, deps brokerStatusDeps) error {
381+ layout, err := offlineLayout(options.offlineRoot)
382+ if err != nil {
383+ if options.ensure {
384+ return ensureOfflineFailure(deps.stdout, brokerStatusReport{Status: "offline"}, err)
385+ }
386+ return err
387+ }
388+ socketPath := brokeripc.SocketPath(layout.Runtime)
389+ offline := brokerStatusReport{Status: "offline", Endpoint: socketPath}
390+
391+ if !options.ensure {
392+ probeCtx, cancel := context.WithTimeout(ctx, options.timeout)
393+ defer cancel()
394+ report, err := deps.probe(probeCtx, socketPath)
395+ if err != nil {
396+ // Dial-only renders the observable state: absence and an
397+ // incompatible live endpoint both read as offline.
398+ return writeBrokerStatusReport(deps.stdout, offline)
399+ }
400+ return writeBrokerStatusReport(deps.stdout, report)
401+ }
402+
403+ // Validate and secure the sandbox root exactly as the launcher does before
404+ // the spawn directory is created or any lock is taken, so status --ensure can
405+ // never operate on paths the launcher would refuse.
406+ if err := safedir.EnsurePrivate(layout.Root); err != nil {
407+ return ensureOfflineFailure(deps.stdout, offline, fmt.Errorf("vev: secure broker sandbox root: %w", err))
408+ }
409+ config, err := brokerconfig.Load(layout)
410+ if err != nil {
411+ return ensureOfflineFailure(deps.stdout, offline, err)
412+ }
413+ grace, graceSet := config.IdleGrace()
414+
415+ deadline := deps.now().Add(options.timeout)
416+ report, err := ensureBrokerReady(ctx, brokerStatusRequest{
417+ layout: layout,
418+ socketPath: socketPath,
419+ deadline: deadline,
420+ timeout: options.timeout,
421+ root: options.offlineRoot,
422+ grace: grace,
423+ graceSet: graceSet,
424+ }, deps)
425+ if err != nil {
426+ return ensureOfflineFailure(deps.stdout, offline, err)
427+ }
428+ return writeBrokerStatusReport(deps.stdout, report)
429+}
430+
431+// ensureOfflineFailure renders one offline report and returns the ensure failure
432+// wrapped in exit code 3, so a layout, configuration, or startup failure always
433+// leaves the caller with the offline JSON document and a distinct exit code.
434+func ensureOfflineFailure(stdout io.Writer, offline brokerStatusReport, err error) error {
435+ if writeErr := writeBrokerStatusReport(stdout, offline); writeErr != nil {
436+ return errors.Join(err, writeErr)
437+ }
438+ return &exitCoded{code: 3, err: err}
439+}
440+
441+// brokerStatusRequest carries one ensure run's fixed inputs.
442+type brokerStatusRequest struct {
443+ layout brokerconfig.Layout
444+ socketPath string
445+ deadline time.Time
446+ timeout time.Duration
447+ root string
448+ grace time.Duration
449+ graceSet bool
450+}
451+
452+// ensureBrokerReady connects to an already-ready broker or elects one spawner
453+// and holds the election until the endpoint answers a valid broker registration
454+// and publishes its initial snapshot, or until the absolute deadline expires.
455+func ensureBrokerReady(ctx context.Context, req brokerStatusRequest, deps brokerStatusDeps) (brokerStatusReport, error) {
456+ ctx, cancel := context.WithDeadline(ctx, req.deadline)
457+ defer cancel()
458+
459+ // Secure the sandbox paths before any lock is taken: the spawn-election
460+ // directory lives beneath the runtime directory, and both must be private
461+ // and symlink-free before they can hold an exclusive lock.
462+ if err := safedir.EnsurePrivate(req.layout.Spawn); err != nil {
463+ return brokerStatusReport{}, fmt.Errorf("vev: secure broker sandbox spawn directory: %w", err)
464+ }
465+ if err := req.layout.VerifyCreated(); err != nil {
466+ return brokerStatusReport{}, fmt.Errorf("vev: verify broker sandbox paths: %w", err)
467+ }
468+
469+ var held brokerStatusLock
470+ defer func() {
471+ if held != nil {
472+ _ = held.Release()
473+ }
474+ }()
475+ spawned := false
476+ backoff := brokerStatusInitialBackoff
477+
478+ for {
479+ if err := ctx.Err(); err != nil {
480+ return brokerStatusReport{}, err
481+ }
482+ if !deps.now().Before(req.deadline) {
483+ return brokerStatusReport{}, fmt.Errorf("%w: endpoint %s did not answer Register within %s", errBrokerNotReady, req.socketPath, req.timeout)
484+ }
485+ report, err := probeWithin(ctx, deps, req.socketPath, req.deadline)
486+ if err == nil {
487+ return report, nil
488+ }
489+ if ctx.Err() != nil {
490+ return brokerStatusReport{}, ctx.Err()
491+ }
492+ // A probe that finished after the deadline is a bounded failure, never
493+ // an absence that would justify one more spawn.
494+ if !deps.now().Before(req.deadline) {
495+ return brokerStatusReport{}, fmt.Errorf("%w: endpoint %s did not answer Register within %s", errBrokerNotReady, req.socketPath, req.timeout)
496+ }
497+ if !errors.Is(err, errBrokerAbsent) {
498+ return brokerStatusReport{}, fmt.Errorf("vev: broker endpoint %s is live but incompatible: %w", req.socketPath, err)
499+ }
500+
501+ if held == nil {
502+ lock, err := deps.acquireElection(req.layout.Spawn)
503+ switch {
504+ case err == nil:
505+ held = lock
506+ // Re-dial after winning the election: the previous holder may
507+ // already have published readiness.
508+ report, err := probeWithin(ctx, deps, req.socketPath, req.deadline)
509+ if err == nil {
510+ return report, nil
511+ }
512+ if ctx.Err() != nil {
513+ return brokerStatusReport{}, ctx.Err()
514+ }
515+ if !errors.Is(err, errBrokerAbsent) {
516+ return brokerStatusReport{}, fmt.Errorf("vev: broker endpoint %s is live but incompatible: %w", req.socketPath, err)
517+ }
518+ case errors.Is(err, lifecycle.ErrBusy):
519+ // Another caller is elected and spawning. Only the elected
520+ // spawner ever probes the broker lifetime lock, so a waiter can
521+ // never briefly steal ownership from the broker it is waiting for.
522+ if err := deps.wait(ctx, backoff); err != nil {
523+ return brokerStatusReport{}, err
524+ }
525+ backoff *= 2
526+ if backoff > brokerStatusMaxBackoff {
527+ backoff = brokerStatusMaxBackoff
528+ }
529+ continue
530+ default:
531+ return brokerStatusReport{}, fmt.Errorf("vev: elect broker spawner: %w", err)
532+ }
533+ }
534+ if held != nil && !spawned {
535+ busy, err := deps.lifetimeOwner(req.layout.Runtime)
536+ if err != nil {
537+ return brokerStatusReport{}, fmt.Errorf("vev: inspect broker lifetime owner: %w", err)
538+ }
539+ if !busy {
540+ if err := deps.spawn(ctx, req.root, req.grace, req.graceSet); err != nil {
541+ return brokerStatusReport{}, err
542+ }
543+ spawned = true
544+ }
545+ }
546+
547+ if err := deps.wait(ctx, backoff); err != nil {
548+ return brokerStatusReport{}, err
549+ }
550+ backoff *= 2
551+ if backoff > brokerStatusMaxBackoff {
552+ backoff = brokerStatusMaxBackoff
553+ }
554+ }
555+}
556+
557+// probeWithin bounds one probe by the smaller of the probe budget and the
558+// remaining overall time.
559+func probeWithin(ctx context.Context, deps brokerStatusDeps, socketPath string, deadline time.Time) (brokerStatusReport, error) {
560+ budget := brokerStatusProbeBudget
561+ if remaining := deadline.Sub(deps.now()); remaining < budget {
562+ budget = remaining
563+ }
564+ if budget <= 0 {
565+ return brokerStatusReport{}, context.DeadlineExceeded
566+ }
567+ probeCtx, cancel := context.WithTimeout(ctx, budget)
568+ defer cancel()
569+ return deps.probe(probeCtx, socketPath)
570+}
571+
572+// probeBrokerStatus is the production probe: it classifies endpoint absence
573+// first, dials with the bounded broker setup, then waits for the broker to
574+// register the connection and publish its initial snapshot.
575+func probeBrokerStatus(ctx context.Context, socketPath string) (brokerStatusReport, error) {
576+ info, err := os.Lstat(socketPath)
577+ switch {
578+ case errors.Is(err, fs.ErrNotExist):
579+ return brokerStatusReport{}, absentError(socketPath)
580+ case err != nil:
581+ return brokerStatusReport{}, fmt.Errorf("vev: inspect broker endpoint %s: %w", socketPath, err)
582+ case info.Mode()&os.ModeSocket == 0:
583+ return brokerStatusReport{}, incompatibleError(socketPath, errors.New("path is not a Unix socket"))
584+ }
585+
586+ service, err := brokeripc.Dial(ctx, socketPath, brokeripc.Config{})
587+ if err != nil {
588+ if backendAbsent(err) {
589+ return brokerStatusReport{}, absentError(socketPath)
590+ }
591+ if ctx.Err() != nil {
592+ return brokerStatusReport{}, ctx.Err()
593+ }
594+ // The endpoint may have retired between the stat and the dial; re-check
595+ // so a broker that just shut down is classified as absent, not as an
596+ // incompatible endpoint that must never be respawned.
597+ if _, statErr := os.Lstat(socketPath); errors.Is(statErr, fs.ErrNotExist) {
598+ return brokerStatusReport{}, absentError(socketPath)
599+ }
600+ return brokerStatusReport{}, incompatibleError(socketPath, err)
601+ }
602+ defer func() { _ = service.Close() }()
603+
604+ sub, err := service.Subscribe()
605+ if err != nil {
606+ return brokerStatusReport{}, incompatibleError(socketPath, err)
607+ }
608+ defer sub.Close()
609+ changed := sub.Changed()
610+ for {
611+ if snapshot := service.Snapshot(); snapshot.Revision != 0 {
612+ return readyBrokerStatus(socketPath, snapshot), nil
613+ }
614+ select {
615+ case <-ctx.Done():
616+ return brokerStatusReport{}, ctx.Err()
617+ case _, ok := <-changed:
618+ if ok {
619+ continue
620+ }
621+ if snapshot := service.Snapshot(); snapshot.Revision != 0 {
622+ return readyBrokerStatus(socketPath, snapshot), nil
623+ }
624+ return brokerStatusReport{}, incompatibleError(socketPath, errors.New("connection ended before the initial snapshot"))
625+ }
626+ }
627+}
628+
629+// readyBrokerStatus projects one committed snapshot into the ready report.
630+func readyBrokerStatus(socketPath string, snapshot ports.BrokerSnapshot) brokerStatusReport {
631+ return brokerStatusReport{
632+ Status: "ready",
633+ Endpoint: socketPath,
634+ Epoch: snapshot.Epoch,
635+ Revision: snapshot.Revision,
636+ HostCount: len(snapshot.Hosts),
637+ }
638+}
639+
640+// backendAbsent reports whether one dial failure means no broker is bound:
641+// a missing socket or a stale socket whose dead owner left the pathname behind.
642+// Any other dial failure is a live endpoint that must not be respawned.
643+func backendAbsent(err error) bool {
644+ var opErr *net.OpError
645+ if !errors.As(err, &opErr) {
646+ return false
647+ }
648+ return errors.Is(err, fs.ErrNotExist) || errors.Is(err, syscall.ENOENT) || errors.Is(err, syscall.ECONNREFUSED)
649+}
650+
651+// absentError wraps one recoverable-absence diagnosis.
652+func absentError(socketPath string) error {
653+ return fmt.Errorf("%w: %s", errBrokerAbsent, socketPath)
654+}
655+
656+// incompatibleError wraps one live-but-incompatible diagnosis.
657+func incompatibleError(socketPath string, cause error) error {
658+ return fmt.Errorf("%w: %s: %w", errBrokerIncompatible, socketPath, cause)
659+}
660+
661+// writeBrokerStatusReport writes exactly one bounded JSON document.
662+func writeBrokerStatusReport(w io.Writer, report brokerStatusReport) error {
663+ data, err := json.Marshal(report)
664+ if err != nil {
665+ return fmt.Errorf("vev: encode broker status: %w", err)
666+ }
667+ if len(data) > brokerStatusMaxReportBytes {
668+ return errors.New("vev: broker status report exceeds the bound")
669+ }
670+ if _, err := w.Write(append(data, '\n')); err != nil {
671+ return fmt.Errorf("vev: write broker status: %w", err)
672+ }
673+ return nil
674+}
A internal/app/broker_offline_status_subprocess_test.go +436 -0
@@ -0,0 +1,436 @@
1+package app
2+
3+import (
4+ "bytes"
5+ "context"
6+ "errors"
7+ "fmt"
8+ "os"
9+ "os/exec"
10+ "path/filepath"
11+ "strconv"
12+ "strings"
13+ "sync"
14+ "syscall"
15+ "testing"
16+ "time"
17+
18+ "github.com/stretchr/testify/require"
19+
20+ "github.com/bnema/vev/internal/adapters/brokerconfig"
21+ "github.com/bnema/vev/internal/adapters/brokeripc"
22+ "github.com/bnema/vev/internal/adapters/lifecycle"
23+)
24+
25+// Real-subprocess coverage for the hidden launcher and status entry points.
26+//
27+// These tests re-execute the test binary through the production dispatch path:
28+// TestMain intercepts the hidden broker commands when brokerHelperEnv is set and
29+// runs app.Run, so `_broker-status` really elects a spawner, `_broker-launcher`
30+// really double-forks, and `_broker-serve` really composes the offline broker.
31+// Every helper records its PID under brokerHelperRecordDirEnv, so a test can
32+// observe how many processes were spawned and terminate every one of them.
33+
34+const (
35+ brokerHelperEnv = "VEV_BROKER_HELPER"
36+ brokerHelperRecordDirEnv = "VEV_BROKER_HELPER_RECORD_DIR"
37+)
38+
39+// Controlled-launcher test env: when set, the hidden `_broker-launcher` helper
40+// records its PID and waits for the release marker instead of running the real
41+// double-fork, so a test can hold a launcher open and prove the spawn wait is
42+// bounded by its context.
43+const (
44+ brokerLauncherBlockRecordEnv = "VEV_BROKER_LAUNCHER_BLOCK_RECORD"
45+ brokerLauncherBlockReleaseEnv = "VEV_BROKER_LAUNCHER_BLOCK_RELEASE"
46+)
47+
48+// recordBrokerHelperProcess appends one helper process's PID, so a test can
49+// discover and later terminate a detached broker or launcher.
50+func recordBrokerHelperProcess(role string) {
51+ dir := os.Getenv(brokerHelperRecordDirEnv)
52+ if dir == "" {
53+ return
54+ }
55+ file, err := os.OpenFile(filepath.Join(dir, role+".pids"), os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o600)
56+ if err != nil {
57+ return
58+ }
59+ defer func() { _ = file.Close() }()
60+ _, _ = fmt.Fprintf(file, "%d\n", os.Getpid())
61+}
62+
63+// shortTempDir creates one temporary directory with a short absolute path.
64+//
65+// The AF_UNIX pathname limit is about 103 bytes, and t.TempDir embeds the full
66+// test name; the broker socket lives several components beneath the sandbox
67+// root, so a long test name would overflow the socket path. A short base keeps
68+// the real subprocess tests honest.
69+func shortTempDir(t *testing.T, prefix string) string {
70+ t.Helper()
71+ dir, err := os.MkdirTemp("", prefix)
72+ require.NoError(t, err)
73+ t.Cleanup(func() { _ = os.RemoveAll(dir) })
74+ return dir
75+}
76+
77+// brokerSubprocessSandbox isolates one real offline root and records every
78+// spawned helper process.
79+type brokerSubprocessSandbox struct {
80+ t *testing.T
81+ root string
82+ layout brokerconfig.Layout
83+ recordDir string
84+ prodRuntime string
85+ prodState string
86+}
87+
88+// newBrokerSubprocessSandbox isolates XDG roots, writes one marker-valid empty
89+// sandbox, and arranges cleanup of every recorded broker and launcher process.
90+func newBrokerSubprocessSandbox(t *testing.T, idleGrace string) *brokerSubprocessSandbox {
91+ t.Helper()
92+ prodRuntime, prodState := t.TempDir(), t.TempDir()
93+ t.Setenv("XDG_RUNTIME_DIR", prodRuntime)
94+ t.Setenv("XDG_STATE_HOME", prodState)
95+ root := filepath.Join(shortTempDir(t, "vevb"), "s")
96+ require.NoError(t, os.MkdirAll(root, 0o700))
97+ writeSandboxConfig(t, root, sandboxEmptyDocument(idleGrace))
98+ recordDir := filepath.Join(shortTempDir(t, "vevr"), "r")
99+ require.NoError(t, os.MkdirAll(recordDir, 0o700))
100+ t.Setenv(brokerHelperEnv, "1")
101+ t.Setenv(brokerHelperRecordDirEnv, recordDir)
102+ layout, err := offlineLayout(root)
103+ require.NoError(t, err)
104+ sandbox := &brokerSubprocessSandbox{t: t, root: root, layout: layout, recordDir: recordDir, prodRuntime: prodRuntime, prodState: prodState}
105+ t.Cleanup(sandbox.cleanup)
106+ return sandbox
107+}
108+
109+// runBrokerCommand executes one helper through the test binary.
110+func runBrokerCommand(t *testing.T, args ...string) (stdout, stderr string, code int) {
111+ t.Helper()
112+ cmd := exec.Command(os.Args[0], args...)
113+ cmd.Env = os.Environ()
114+ var out, errb bytes.Buffer
115+ cmd.Stdout = &out
116+ cmd.Stderr = &errb
117+ err := cmd.Run()
118+ if err != nil {
119+ var exitErr *exec.ExitError
120+ if !errors.As(err, &exitErr) {
121+ t.Fatalf("run broker command %v: %v", args, err)
122+ }
123+ code = exitErr.ExitCode()
124+ }
125+ return out.String(), errb.String(), code
126+}
127+
128+// status runs `_broker-status` for this sandbox with the supplied extra flags.
129+func (s *brokerSubprocessSandbox) status(extra ...string) (stdout, stderr string, code int) {
130+ s.t.Helper()
131+ args := append([]string{brokerStatusCommand, "--offline-root", s.root}, extra...)
132+ return runBrokerCommand(s.t, args...)
133+}
134+
135+// socketPath is the broker endpoint inside the sandbox runtime directory.
136+func (s *brokerSubprocessSandbox) socketPath() string {
137+ return brokeripc.SocketPath(s.layout.Runtime)
138+}
139+
140+// recorded returns every PID a role recorded, in spawn order.
141+func (s *brokerSubprocessSandbox) recorded(role string) []int {
142+ s.t.Helper()
143+ raw, err := os.ReadFile(filepath.Join(s.recordDir, role+".pids"))
144+ if errors.Is(err, os.ErrNotExist) {
145+ return nil
146+ }
147+ require.NoError(s.t, err)
148+ var pids []int
149+ for _, field := range strings.Fields(string(raw)) {
150+ pid, convErr := strconv.Atoi(field)
151+ require.NoError(s.t, convErr)
152+ pids = append(pids, pid)
153+ }
154+ return pids
155+}
156+
157+// live returns the recorded PIDs that still exist.
158+func (s *brokerSubprocessSandbox) live(role string) []int {
159+ s.t.Helper()
160+ var live []int
161+ for _, pid := range s.recorded(role) {
162+ if err := syscall.Kill(pid, 0); err == nil || errors.Is(err, syscall.EPERM) {
163+ live = append(live, pid)
164+ }
165+ }
166+ return live
167+}
168+
169+// terminate signals every recorded PID of one role and waits for each exit.
170+func (s *brokerSubprocessSandbox) terminate(role string, signal syscall.Signal) {
171+ s.t.Helper()
172+ for _, pid := range s.recorded(role) {
173+ if err := syscall.Kill(pid, signal); err != nil && !errors.Is(err, syscall.ESRCH) {
174+ s.t.Errorf("signal %v pid %d: %v", signal, pid, err)
175+ continue
176+ }
177+ require.NoError(s.t, waitForProcessExit(pid, 10*time.Second))
178+ }
179+}
180+
181+// waitForSocketGone waits until the broker endpoint no longer exists.
182+func (s *brokerSubprocessSandbox) waitForSocketGone(timeout time.Duration) bool {
183+ deadline := time.Now().Add(timeout)
184+ for time.Now().Before(deadline) {
185+ if _, err := os.Lstat(s.socketPath()); errors.Is(err, os.ErrNotExist) {
186+ return true
187+ }
188+ time.Sleep(20 * time.Millisecond)
189+ }
190+ return false
191+}
192+
193+// cleanup terminates every recorded broker and launcher so no test leaks an
194+// orphan process.
195+func (s *brokerSubprocessSandbox) cleanup() {
196+ for _, role := range []string{brokerServeCommand, brokerLauncherCommand} {
197+ for _, pid := range s.recorded(role) {
198+ if err := syscall.Kill(pid, syscall.SIGKILL); err != nil && !errors.Is(err, syscall.ESRCH) {
199+ continue
200+ }
201+ _ = waitForProcessExit(pid, 10*time.Second)
202+ }
203+ }
204+}
205+
206+// TestBrokerStatusSubprocessElectsExactlyOneBroker races several real status
207+// processes against one socket: exactly one broker process, one lifetime owner,
208+// and one epoch must result.
209+func TestBrokerStatusSubprocessElectsExactlyOneBroker(t *testing.T) {
210+ sandbox := newBrokerSubprocessSandbox(t, "5s")
211+ const racers = 4
212+
213+ type outcome struct {
214+ stdout, stderr string
215+ code int
216+ }
217+ results := make([]outcome, racers)
218+ var wg sync.WaitGroup
219+ for i := 0; i < racers; i++ {
220+ wg.Add(1)
221+ go func(index int) {
222+ defer wg.Done()
223+ stdout, stderr, code := runBrokerCommand(t, brokerStatusCommand, "--offline-root", sandbox.root, "--ensure", "--timeout", "20s")
224+ results[index] = outcome{stdout: stdout, stderr: stderr, code: code}
225+ }(i)
226+ }
227+ wg.Wait()
228+
229+ epochs := make(map[uint64]struct{}, racers)
230+ for i, result := range results {
231+ require.Equalf(t, 0, result.code, "racer %d exited %d: %s", i, result.code, result.stderr)
232+ report := decodeOneReport(t, result.stdout)
233+ require.Equal(t, "ready", report.Status)
234+ require.Equal(t, sandbox.socketPath(), report.Endpoint)
235+ epochs[uint64(report.Epoch)] = struct{}{}
236+ }
237+ require.Len(t, epochs, 1, "every racer must observe the same broker epoch")
238+ require.Len(t, sandbox.recorded(brokerServeCommand), 1, "exactly one broker process must ever be spawned")
239+ require.Len(t, sandbox.live(brokerServeCommand), 1, "exactly one broker process must be live")
240+ requireProductionUntouched(t, sandbox.prodRuntime, sandbox.prodState)
241+}
242+
243+// TestBrokerStatusSubprocessIdleExitRestartsWithNewEpoch lets the broker idle
244+// out, then restarts it on demand and proves the new process samples a fresh
245+// epoch.
246+func TestBrokerStatusSubprocessIdleExitRestartsWithNewEpoch(t *testing.T) {
247+ sandbox := newBrokerSubprocessSandbox(t, "2s")
248+
249+ stdout, stderr, code := sandbox.status("--ensure", "--timeout", "20s")
250+ require.Equal(t, 0, code, stderr)
251+ first := decodeOneReport(t, stdout)
252+ require.Equal(t, "ready", first.Status)
253+ require.Positive(t, first.Epoch)
254+
255+ require.True(t, sandbox.waitForSocketGone(15*time.Second), "the idle broker must shut itself down")
256+ for _, pid := range sandbox.recorded(brokerServeCommand) {
257+ require.NoError(t, waitForProcessExit(pid, 10*time.Second), "the idle broker must exit")
258+ }
259+ require.Empty(t, sandbox.live(brokerServeCommand), "the idle broker must exit")
260+
261+ stdout, stderr, code = sandbox.status("--ensure", "--timeout", "20s")
262+ require.Equal(t, 0, code, stderr)
263+ second := decodeOneReport(t, stdout)
264+ require.Equal(t, "ready", second.Status)
265+ require.NotEqual(t, first.Epoch, second.Epoch, "an on-demand restart must sample a new epoch")
266+ require.Len(t, sandbox.recorded(brokerServeCommand), 2, "one broker per ensure is expected")
267+ require.Len(t, sandbox.live(brokerServeCommand), 1, "only the restarted broker may remain")
268+}
269+
270+// TestBrokerStatusSubprocessRecoversStaleSocket proves a stale socket left by a
271+// dead owner is recoverable absence, not an incompatible endpoint.
272+func TestBrokerStatusSubprocessRecoversStaleSocket(t *testing.T) {
273+ sandbox := newBrokerSubprocessSandbox(t, "3s")
274+ require.NoError(t, os.MkdirAll(sandbox.layout.Runtime, 0o700))
275+ bindStaleUnixSocket(t, sandbox.socketPath())
276+
277+ stdout, stderr, code := sandbox.status("--ensure", "--timeout", "20s")
278+ require.Equal(t, 0, code, stderr)
279+ report := decodeOneReport(t, stdout)
280+ require.Equal(t, "ready", report.Status)
281+ require.Len(t, sandbox.live(brokerServeCommand), 1)
282+}
283+
284+// TestBrokerStatusSubprocessWaitsForLiveSpawnerThenRecovers holds the
285+// descriptor-backed election lock like a spawner that died mid-startup: the
286+// waiter must not spawn, and must take over once the lock is released.
287+func TestBrokerStatusSubprocessWaitsForLiveSpawnerThenRecovers(t *testing.T) {
288+ sandbox := newBrokerSubprocessSandbox(t, "3s")
289+
290+ spawner, err := lifecycle.TryAcquire(sandbox.layout.Spawn)
291+ require.NoError(t, err)
292+
293+ stdout, _, code := sandbox.status("--ensure", "--timeout", "500ms")
294+ require.Equal(t, 3, code)
295+ require.Equal(t, "offline", decodeOneReport(t, stdout).Status)
296+ require.Empty(t, sandbox.recorded(brokerServeCommand), "a waiter must not spawn while another spawner is live")
297+
298+ // The kernel releases the spawner's descriptor when it dies; releasing here
299+ // is exactly that event.
300+ require.NoError(t, spawner.Release())
301+ stdout, stderr, code := sandbox.status("--ensure", "--timeout", "20s")
302+ require.Equal(t, 0, code, stderr)
303+ require.Equal(t, "ready", decodeOneReport(t, stdout).Status)
304+ require.Len(t, sandbox.live(brokerServeCommand), 1)
305+}
306+
307+// TestBrokerStatusSubprocessSilentEndpointFailsWithoutSpawning proves a live
308+// endpoint that never answers the broker handshake is diagnosed as
309+// incompatible under the probe budget, and never respawned over.
310+func TestBrokerStatusSubprocessSilentEndpointFailsWithoutSpawning(t *testing.T) {
311+ sandbox := newBrokerSubprocessSandbox(t, "3s")
312+ require.NoError(t, os.MkdirAll(sandbox.layout.Runtime, 0o700))
313+ // A listener that accepts and never answers: the broker preamble stalls.
314+ _ = bindUnixSocket(t, sandbox.socketPath())
315+
316+ // The timeout must exceed the probe budget so the stall is classified as a
317+ // live-but-incompatible endpoint instead of exhausting the overall deadline.
318+ start := time.Now()
319+ stdout, stderr, code := sandbox.status("--ensure", "--timeout", "5s")
320+ require.Equal(t, 3, code)
321+ require.Equal(t, "offline", decodeOneReport(t, stdout).Status)
322+ require.Contains(t, stderr, "live but incompatible", "a silent live endpoint must be classified incompatible")
323+ require.Less(t, time.Since(start), 5*time.Second, "the probe budget must classify a silent endpoint before the overall deadline")
324+ require.Empty(t, sandbox.recorded(brokerServeCommand), "a live-but-silent endpoint must not be respawned over")
325+}
326+
327+// TestBrokerLauncherSpawnWaitIsBoundedByContext proves a controlled blocking
328+// launcher is terminated when the spawn context is cancelled, so the overall
329+// deadline/signal bounds the launcher wait. The helper never starts a detached
330+// serve, so this is purely the bounded-wait half; other subprocess tests prove a
331+// successful launcher's detached serve survives.
332+func TestBrokerLauncherSpawnWaitIsBoundedByContext(t *testing.T) {
333+ record := filepath.Join(shortTempDir(t, "vevl"), "launcher.pid")
334+ release := filepath.Join(t.TempDir(), "release")
335+ t.Setenv(brokerHelperEnv, "1")
336+ t.Setenv(brokerLauncherBlockRecordEnv, record)
337+ t.Setenv(brokerLauncherBlockReleaseEnv, release)
338+ t.Cleanup(func() { _ = os.WriteFile(release, nil, 0o600) })
339+
340+ ctx, cancel := context.WithCancel(context.Background())
341+ defer cancel()
342+ result := make(chan error, 1)
343+ go func() { result <- spawnBrokerLauncher(ctx, t.TempDir(), 0, false) }()
344+
345+ pid := waitForProcessRecord(t, record)
346+ select {
347+ case err := <-result:
348+ t.Fatalf("spawnBrokerLauncher returned while the controlled launcher %d was still running: %v", pid, err)
349+ default:
350+ }
351+
352+ cancel()
353+ select {
354+ case err := <-result:
355+ require.Error(t, err, "cancelling the context must abort the launcher wait")
356+ case <-time.After(5 * time.Second):
357+ t.Fatal("cancelling the context must bound the launcher spawn wait")
358+ }
359+ require.NoError(t, waitForProcessExit(pid, 5*time.Second), "CommandContext must terminate a blocking launcher")
360+}
361+
362+// TestBrokerStatusSubprocessLauncherDiagnosesConflicts proves an explicit
363+// launcher grace that conflicts with the provisioned one is refused before any
364+// process is spawned, and that a missing configuration fails the launcher.
365+func TestBrokerStatusSubprocessLauncherDiagnosesConflicts(t *testing.T) {
366+ t.Run("conflicting idle grace", func(t *testing.T) {
367+ sandbox := newBrokerSubprocessSandbox(t, "90s")
368+ stdout, stderr, code := runBrokerCommand(t, brokerLauncherCommand, "--offline-root", sandbox.root, "--idle-grace", "1m")
369+ require.NotEqual(t, 0, code)
370+ require.Contains(t, stderr, "conflicts with the provisioned idle grace")
371+ require.Empty(t, strings.TrimSpace(stdout))
372+ require.Empty(t, sandbox.recorded(brokerServeCommand))
373+ })
374+
375+ t.Run("missing configuration", func(t *testing.T) {
376+ prodRuntime, prodState := t.TempDir(), t.TempDir()
377+ t.Setenv("XDG_RUNTIME_DIR", prodRuntime)
378+ t.Setenv("XDG_STATE_HOME", prodState)
379+ recordDir := filepath.Join(t.TempDir(), "records")
380+ require.NoError(t, os.MkdirAll(recordDir, 0o700))
381+ t.Setenv(brokerHelperEnv, "1")
382+ t.Setenv(brokerHelperRecordDirEnv, recordDir)
383+ root := filepath.Join(shortTempDir(t, "vevx"), "s")
384+ require.NoError(t, os.MkdirAll(root, 0o700))
385+
386+ _, stderr, code := runBrokerCommand(t, brokerLauncherCommand, "--offline-root", root)
387+ require.NotEqual(t, 0, code)
388+ require.NotEmpty(t, strings.TrimSpace(stderr))
389+ require.Empty(t, recordedPIDsIn(t, recordDir, brokerServeCommand))
390+ })
391+}
392+
393+// TestBrokerStatusSubprocessNoOrphanProcess proves the launcher exits on its
394+// own, one broker remains, and terminating it leaves no recorded process live.
395+func TestBrokerStatusSubprocessNoOrphanProcess(t *testing.T) {
396+ sandbox := newBrokerSubprocessSandbox(t, "5s")
397+
398+ stdout, stderr, code := sandbox.status("--ensure", "--timeout", "20s")
399+ require.Equal(t, 0, code, stderr)
400+ require.Equal(t, "ready", decodeOneReport(t, stdout).Status)
401+
402+ require.NotEmpty(t, sandbox.recorded(brokerLauncherCommand))
403+ for _, pid := range sandbox.recorded(brokerLauncherCommand) {
404+ require.NoError(t, waitForProcessExit(pid, 5*time.Second), "the short-lived launcher must exit")
405+ }
406+ require.Len(t, sandbox.live(brokerServeCommand), 1)
407+
408+ _, err := lifecycle.TryAcquire(sandbox.layout.Runtime)
409+ require.ErrorIs(t, err, lifecycle.ErrBusy, "the live broker must own the lifetime lock")
410+
411+ sandbox.terminate(brokerServeCommand, syscall.SIGTERM)
412+ require.Empty(t, sandbox.live(brokerLauncherCommand), "no launcher may be left behind")
413+ require.Empty(t, sandbox.live(brokerServeCommand), "no broker may be left behind")
414+ require.True(t, sandbox.waitForSocketGone(5*time.Second), "an orderly broker shutdown must unlink its socket")
415+ owner, err := lifecycle.TryAcquire(sandbox.layout.Runtime)
416+ require.NoError(t, err, "the terminated broker must release the lifetime lock")
417+ require.NoError(t, owner.Release())
418+ requireProductionUntouched(t, sandbox.prodRuntime, sandbox.prodState)
419+}
420+
421+// recordedPIDsIn reads one role's PID record from an arbitrary directory.
422+func recordedPIDsIn(t *testing.T, dir, role string) []int {
423+ t.Helper()
424+ raw, err := os.ReadFile(filepath.Join(dir, role+".pids"))
425+ if errors.Is(err, os.ErrNotExist) {
426+ return nil
427+ }
428+ require.NoError(t, err)
429+ var pids []int
430+ for _, field := range strings.Fields(string(raw)) {
431+ pid, convErr := strconv.Atoi(field)
432+ require.NoError(t, convErr)
433+ pids = append(pids, pid)
434+ }
435+ return pids
436+}
A internal/app/broker_offline_status_test.go +684 -0
@@ -0,0 +1,684 @@
1+package app
2+
3+import (
4+ "bytes"
5+ "context"
6+ "encoding/json"
7+ "errors"
8+ "net"
9+ "os"
10+ "path/filepath"
11+ "strings"
12+ "sync/atomic"
13+ "testing"
14+ "time"
15+
16+ "github.com/stretchr/testify/require"
17+
18+ "github.com/bnema/vev/internal/adapters/brokerconfig"
19+ "github.com/bnema/vev/internal/adapters/brokeripc"
20+ "github.com/bnema/vev/internal/adapters/lifecycle"
21+)
22+
23+// sandboxEmptyDocument is a marker-valid sandbox configuration with no
24+// registrations and an optional provisioned idle grace.
25+func sandboxEmptyDocument(idleGrace string) map[string]any {
26+ document := map[string]any{"marker": brokerconfig.Marker, "registrations": []any{}}
27+ if idleGrace != "" {
28+ document["idleGrace"] = idleGrace
29+ }
30+ return document
31+}
32+
33+// emptySandboxRoot writes one marker-valid sandbox root and returns its layout.
34+func emptySandboxRoot(t *testing.T, idleGrace string) brokerconfig.Layout {
35+ t.Helper()
36+ isolateSandboxEnv(t)
37+ root := filepath.Join(shortTempDir(t, "vevu"), "s")
38+ require.NoError(t, os.MkdirAll(root, 0o700))
39+ writeSandboxConfig(t, root, sandboxEmptyDocument(idleGrace))
40+ layout, err := offlineLayout(root)
41+ require.NoError(t, err)
42+ return layout
43+}
44+
45+// fakeStatusLock records releases for one held election.
46+type fakeStatusLock struct{ releases atomic.Int32 }
47+
48+func (l *fakeStatusLock) Release() error {
49+ l.releases.Add(1)
50+ return nil
51+}
52+
53+// readyReport is the report returned when a scripted probe succeeds.
54+func readyReport(socketPath string) brokerStatusReport {
55+ return brokerStatusReport{Status: "ready", Endpoint: socketPath, Epoch: 7, Revision: 3, HostCount: 2}
56+}
57+
58+// testStatusDeps builds deterministic status seams. Virtual time starts at the
59+// real instant and only advances when the loop waits, so a short timeout
60+// exhausts without real sleeping while the production context deadline stays
61+// comfortably in the future. The default probe always reports absence.
62+func testStatusDeps(t *testing.T) (brokerStatusDeps, *statusRecorder) {
63+ t.Helper()
64+ rec := &statusRecorder{
65+ now: time.Now(),
66+ stdout: &bytes.Buffer{},
67+ }
68+ deps := brokerStatusDeps{
69+ now: func() time.Time { return rec.now },
70+ wait: func(ctx context.Context, d time.Duration) error {
71+ rec.now = rec.now.Add(d)
72+ return ctx.Err()
73+ },
74+ acquireElection: func(string) (brokerStatusLock, error) {
75+ rec.elections.Add(1)
76+ lock := &fakeStatusLock{}
77+ rec.locks = append(rec.locks, lock)
78+ return lock, nil
79+ },
80+ lifetimeOwner: func(string) (bool, error) { return false, nil },
81+ spawn: func(context.Context, string, time.Duration, bool) error {
82+ rec.spawns.Add(1)
83+ return nil
84+ },
85+ stdout: rec.stdout,
86+ }
87+ rec.setProbe(&deps, func(_ int, socketPath string) (brokerStatusReport, error) {
88+ return brokerStatusReport{}, absentError(socketPath)
89+ })
90+ return deps, rec
91+}
92+
93+// statusRecorder captures one status run's observable effects.
94+type statusRecorder struct {
95+ now time.Time
96+ probes atomic.Int32
97+ spawns atomic.Int32
98+ elections atomic.Int32
99+ locks []*fakeStatusLock
100+ stdout *bytes.Buffer
101+}
102+
103+// setProbe installs one scripted probe that sees its 1-based attempt number.
104+func (r *statusRecorder) setProbe(deps *brokerStatusDeps, probe func(attempt int, socketPath string) (brokerStatusReport, error)) {
105+ deps.probe = func(_ context.Context, socketPath string) (brokerStatusReport, error) {
106+ return probe(int(r.probes.Add(1)), socketPath)
107+ }
108+}
109+
110+func (r *statusRecorder) report(t *testing.T) brokerStatusReport {
111+ t.Helper()
112+ var report brokerStatusReport
113+ require.NoError(t, json.Unmarshal([]byte(strings.TrimSpace(r.stdout.String())), &report))
114+ return report
115+}
116+
117+// decodeOneReport asserts stdout holds exactly one JSON document and decodes it.
118+func decodeOneReport(t *testing.T, raw string) brokerStatusReport {
119+ t.Helper()
120+ trimmed := strings.TrimSpace(raw)
121+ require.Equal(t, 1, strings.Count(trimmed, "\n")+1, "stdout must hold exactly one report line")
122+ var report brokerStatusReport
123+ require.NoError(t, json.Unmarshal([]byte(trimmed), &report))
124+ return report
125+}
126+
127+func TestParseBrokerLauncherArgs(t *testing.T) {
128+ tests := []struct {
129+ name string
130+ args []string
131+ want brokerLauncherOptions
132+ wantErr string
133+ }{
134+ {name: "root only", args: []string{"--offline-root", "/srv/sandbox"}, want: brokerLauncherOptions{offlineRoot: "/srv/sandbox"}},
135+ {
136+ name: "root and grace",
137+ args: []string{"--offline-root", "/srv/sandbox", "--idle-grace", "90s"},
138+ want: brokerLauncherOptions{offlineRoot: "/srv/sandbox", idleGrace: 90 * time.Second},
139+ },
140+ {name: "missing root", args: nil, wantErr: "requires --offline-root"},
141+ {name: "missing root value", args: []string{"--offline-root"}, wantErr: "requires a path"},
142+ {name: "duplicate root", args: []string{"--offline-root", "/a", "--offline-root", "/b"}, wantErr: "duplicate --offline-root"},
143+ {name: "duplicate grace", args: []string{"--offline-root", "/a", "--idle-grace", "1s", "--idle-grace", "2s"}, wantErr: "duplicate --idle-grace"},
144+ {name: "zero grace", args: []string{"--offline-root", "/a", "--idle-grace", "0s"}, wantErr: "must be positive"},
145+ {name: "invalid grace", args: []string{"--offline-root", "/a", "--idle-grace", "soon"}, wantErr: "not a duration"},
146+ {name: "ensure is not a launcher flag", args: []string{"--offline-root", "/a", "--ensure"}, wantErr: "unknown flag"},
147+ {name: "positional", args: []string{"--offline-root", "/a", "extra"}, wantErr: "positional"},
148+ }
149+ for _, tt := range tests {
150+ t.Run(tt.name, func(t *testing.T) {
151+ command, err := parseBrokerLauncherArgs(tt.args)
152+ if tt.wantErr != "" {
153+ require.Error(t, err)
154+ require.Contains(t, err.Error(), tt.wantErr)
155+ return
156+ }
157+ require.NoError(t, err)
158+ require.Equal(t, kindBrokerLauncher, command.kind)
159+ require.Equal(t, tt.want, command.brokerLauncher)
160+ })
161+ }
162+}
163+
164+func TestParseBrokerStatusArgs(t *testing.T) {
165+ tests := []struct {
166+ name string
167+ args []string
168+ want brokerStatusOptions
169+ wantErr string
170+ }{
171+ {
172+ name: "root only defaults the timeout",
173+ args: []string{"--offline-root", "/srv/sandbox"},
174+ want: brokerStatusOptions{offlineRoot: "/srv/sandbox", timeout: brokerStatusDefaultTimeout},
175+ },
176+ {
177+ name: "ensure and timeout",
178+ args: []string{"--offline-root", "/a", "--ensure", "--timeout", "250ms"},
179+ want: brokerStatusOptions{offlineRoot: "/a", ensure: true, timeout: 250 * time.Millisecond},
180+ },
181+ {name: "missing root", args: []string{"--ensure"}, wantErr: "requires --offline-root"},
182+ {name: "duplicate root", args: []string{"--offline-root", "/a", "--offline-root", "/b"}, wantErr: "duplicate --offline-root"},
183+ {name: "duplicate ensure", args: []string{"--offline-root", "/a", "--ensure", "--ensure"}, wantErr: "duplicate --ensure"},
184+ {name: "duplicate timeout", args: []string{"--offline-root", "/a", "--timeout", "1s", "--timeout", "2s"}, wantErr: "duplicate --timeout"},
185+ {name: "missing timeout value", args: []string{"--offline-root", "/a", "--timeout"}, wantErr: "requires a duration"},
186+ {name: "zero timeout", args: []string{"--offline-root", "/a", "--timeout", "0s"}, wantErr: "must be positive"},
187+ {name: "invalid timeout", args: []string{"--offline-root", "/a", "--timeout", "soon"}, wantErr: "not a duration"},
188+ {name: "idle grace is not a status flag", args: []string{"--offline-root", "/a", "--idle-grace", "1s"}, wantErr: "unknown flag"},
189+ {name: "positional", args: []string{"--offline-root", "/a", "extra"}, wantErr: "positional"},
190+ }
191+ for _, tt := range tests {
192+ t.Run(tt.name, func(t *testing.T) {
193+ command, err := parseBrokerStatusArgs(tt.args)
194+ if tt.wantErr != "" {
195+ require.Error(t, err)
196+ require.Contains(t, err.Error(), tt.wantErr)
197+ return
198+ }
199+ require.NoError(t, err)
200+ require.Equal(t, kindBrokerStatus, command.kind)
201+ require.Equal(t, tt.want, command.brokerStatus)
202+ })
203+ }
204+}
205+
206+func TestBrokerLauncherAndStatusAreHiddenFromPublicHelp(t *testing.T) {
207+ require.NotContains(t, usageText, brokerLauncherCommand)
208+ require.NotContains(t, usageText, brokerStatusCommand)
209+ _, err := parseArgs([]string{brokerLauncherCommand, "--offline-root", "/srv/sandbox"})
210+ require.NoError(t, err)
211+ _, err = parseArgs([]string{brokerStatusCommand, "--offline-root", "/srv/sandbox", "--ensure"})
212+ require.NoError(t, err)
213+ _, err = parseArgs([]string{"--help"})
214+ require.NoError(t, err)
215+}
216+
217+func TestEffectiveIdleGrace(t *testing.T) {
218+ layout := emptySandboxRoot(t, "90s")
219+ config, err := brokerconfig.Load(layout)
220+ require.NoError(t, err)
221+ unprovisioned := emptySandboxRoot(t, "")
222+ plain, err := brokerconfig.Load(unprovisioned)
223+ require.NoError(t, err)
224+
225+ tests := []struct {
226+ name string
227+ config *brokerconfig.Config
228+ explicit time.Duration
229+ want time.Duration
230+ wantErr string
231+ }{
232+ {name: "unset uses provisioned", config: config, want: 90 * time.Second},
233+ {name: "matching explicit is accepted", config: config, explicit: 90 * time.Second, want: 90 * time.Second},
234+ {name: "conflicting explicit is diagnosed", config: config, explicit: time.Minute, wantErr: "conflicts with the provisioned idle grace"},
235+ {name: "unprovisioned explicit wins", config: plain, explicit: 42 * time.Second, want: 42 * time.Second},
236+ {name: "unprovisioned unset defers to the default", config: plain, want: 0},
237+ }
238+ for _, tt := range tests {
239+ t.Run(tt.name, func(t *testing.T) {
240+ got, err := effectiveIdleGrace(tt.config, tt.explicit)
241+ if tt.wantErr != "" {
242+ require.Error(t, err)
243+ require.Contains(t, err.Error(), tt.wantErr)
244+ return
245+ }
246+ require.NoError(t, err)
247+ require.Equal(t, tt.want, got)
248+ })
249+ }
250+}
251+
252+func TestBrokerServeRefusesConflictingIdleGrace(t *testing.T) {
253+ layout := emptySandboxRoot(t, "90s")
254+ ctx, cancel := context.WithCancel(context.Background())
255+ defer cancel()
256+ deps, _ := testBrokerServeDeps(newSandboxClock())
257+ err := runBrokerServe(ctx, brokerServeOptions{offlineRoot: layout.Root, idleGrace: time.Minute}, deps)
258+ require.Error(t, err)
259+ require.Contains(t, err.Error(), "conflicts with the provisioned idle grace")
260+ require.NoFileExists(t, filepath.Join(layout.Runtime, "lifecycle.lock"), "a refused serve must not take the lifetime lock")
261+}
262+
263+func TestBrokerLauncherRefusesConflictingIdleGraceBeforeSpawning(t *testing.T) {
264+ layout := emptySandboxRoot(t, "90s")
265+ err := runBrokerLauncher(brokerLauncherOptions{offlineRoot: layout.Root, idleGrace: time.Minute})
266+ require.Error(t, err)
267+ require.Contains(t, err.Error(), "conflicts with the provisioned idle grace")
268+}
269+
270+func TestBrokerStatusDialOnlyReportsOfflineWithoutCreatingAnything(t *testing.T) {
271+ layout := emptySandboxRoot(t, "")
272+ deps := defaultBrokerStatusDeps()
273+ var out bytes.Buffer
274+ deps.stdout = &out
275+
276+ require.NoError(t, runBrokerStatus(context.Background(), brokerStatusOptions{offlineRoot: layout.Root, timeout: time.Second}, deps))
277+ report := decodeOneReport(t, out.String())
278+ require.Equal(t, "offline", report.Status)
279+ require.Equal(t, brokeripc.SocketPath(layout.Runtime), report.Endpoint)
280+ require.Zero(t, report.Epoch)
281+ require.Zero(t, report.Revision)
282+ require.Zero(t, report.HostCount)
283+
284+ require.NoDirExists(t, layout.Runtime, "dial-only status must not create the runtime directory")
285+ require.NoDirExists(t, layout.Spawn, "dial-only status must not create the spawn directory")
286+}
287+
288+func TestBrokerStatusDialOnlyReadyAgainstLiveSandbox(t *testing.T) {
289+ root, prodRuntime, prodState := isolateSandboxEnv(t)
290+ fixture := newBrokerMuxFixture(t, brokerTestPolicy())
291+ require.NoError(t, os.MkdirAll(root, 0o700))
292+ writeSandboxConfig(t, root, sandboxRegistrationDocument(fixture.route))
293+
294+ clk := newSandboxClock()
295+ serveDeps, ready := testBrokerServeDeps(clk)
296+ ctx, cancel := context.WithCancel(context.Background())
297+ defer cancel()
298+ done := runSandbox(ctx, brokerServeOptions{offlineRoot: root}, serveDeps)
299+ awaitSandboxReady(t, ready)
300+
301+ deps := defaultBrokerStatusDeps()
302+ var out bytes.Buffer
303+ deps.stdout = &out
304+ require.NoError(t, runBrokerStatus(context.Background(), brokerStatusOptions{offlineRoot: root, timeout: time.Second}, deps))
305+ report := decodeOneReport(t, out.String())
306+ require.Equal(t, "ready", report.Status)
307+ require.Positive(t, report.Epoch)
308+ require.Positive(t, report.Revision)
309+ // The sandbox registry restores from its own empty durable store; the
310+ // provisioned endpoints feed the resolver, not the host inventory.
311+ require.Zero(t, report.HostCount)
312+
313+ awaitSandboxCancel(t, cancel, done)
314+ requireProductionUntouched(t, prodRuntime, prodState)
315+}
316+
317+func TestBrokerStatusEnsureSpawnsOnceThenReportsReady(t *testing.T) {
318+ layout := emptySandboxRoot(t, "")
319+ deps, rec := testStatusDeps(t)
320+ rec.setProbe(&deps, func(_ int, socketPath string) (brokerStatusReport, error) {
321+ if rec.spawns.Load() > 0 {
322+ return readyReport(socketPath), nil
323+ }
324+ return brokerStatusReport{}, absentError(socketPath)
325+ })
326+ require.NoError(t, runBrokerStatus(context.Background(), brokerStatusOptions{offlineRoot: layout.Root, ensure: true, timeout: time.Second}, deps))
327+
328+ require.Equal(t, int32(1), rec.spawns.Load(), "exactly one spawn is expected")
329+ require.Equal(t, int32(1), rec.elections.Load(), "exactly one election is expected")
330+ require.Len(t, rec.locks, 1)
331+ require.Equal(t, int32(1), rec.locks[0].releases.Load(), "the election must be released after readiness")
332+ require.Equal(t, "ready", rec.report(t).Status)
333+}
334+
335+func TestBrokerStatusEnsureWaitsForAnotherSpawner(t *testing.T) {
336+ layout := emptySandboxRoot(t, "")
337+ var readyAfter int32 = 4
338+ deps, rec := testStatusDeps(t)
339+ rec.setProbe(&deps, func(attempt int, socketPath string) (brokerStatusReport, error) {
340+ if int32(attempt) >= readyAfter {
341+ return readyReport(socketPath), nil
342+ }
343+ return brokerStatusReport{}, absentError(socketPath)
344+ })
345+ deps.acquireElection = func(string) (brokerStatusLock, error) {
346+ rec.elections.Add(1)
347+ return nil, lifecycle.ErrBusy
348+ }
349+ require.NoError(t, runBrokerStatus(context.Background(), brokerStatusOptions{offlineRoot: layout.Root, ensure: true, timeout: time.Second}, deps))
350+
351+ require.Zero(t, rec.spawns.Load(), "a waiter must never spawn while another caller is elected")
352+ require.Positive(t, rec.elections.Load())
353+ require.Equal(t, "ready", rec.report(t).Status)
354+}
355+
356+func TestBrokerStatusEnsureDefersToLifetimeOwnerBeforeSpawning(t *testing.T) {
357+ layout := emptySandboxRoot(t, "")
358+ var ownerCalls int32
359+ deps, rec := testStatusDeps(t)
360+ rec.setProbe(&deps, func(_ int, socketPath string) (brokerStatusReport, error) {
361+ if rec.spawns.Load() > 0 {
362+ return readyReport(socketPath), nil
363+ }
364+ return brokerStatusReport{}, absentError(socketPath)
365+ })
366+ deps.lifetimeOwner = func(string) (bool, error) {
367+ if atomic.AddInt32(&ownerCalls, 1) <= 3 {
368+ return true, nil
369+ }
370+ return false, nil
371+ }
372+ require.NoError(t, runBrokerStatus(context.Background(), brokerStatusOptions{offlineRoot: layout.Root, ensure: true, timeout: time.Second}, deps))
373+ require.Equal(t, int32(1), rec.spawns.Load(), "a live lifetime owner must suppress the spawn until it is gone")
374+ require.Equal(t, int32(1), rec.elections.Load(), "the election is taken only after the live owner releases the sandbox")
375+}
376+
377+func TestBrokerStatusEnsureRecoversAfterSpawnerDeath(t *testing.T) {
378+ layout := emptySandboxRoot(t, "")
379+ var electionAttempts int32
380+ deps, rec := testStatusDeps(t)
381+ rec.setProbe(&deps, func(_ int, socketPath string) (brokerStatusReport, error) {
382+ if rec.spawns.Load() > 0 {
383+ return readyReport(socketPath), nil
384+ }
385+ return brokerStatusReport{}, absentError(socketPath)
386+ })
387+ deps.acquireElection = func(string) (brokerStatusLock, error) {
388+ rec.elections.Add(1)
389+ if atomic.AddInt32(&electionAttempts, 1) <= 3 {
390+ // The previous elected spawner is still alive (or mid-startup).
391+ return nil, lifecycle.ErrBusy
392+ }
393+ lock := &fakeStatusLock{}
394+ rec.locks = append(rec.locks, lock)
395+ return lock, nil
396+ }
397+ require.NoError(t, runBrokerStatus(context.Background(), brokerStatusOptions{offlineRoot: layout.Root, ensure: true, timeout: time.Second}, deps))
398+ require.Equal(t, int32(1), rec.spawns.Load(), "a waiter must take over once the elected spawner dies")
399+ require.Equal(t, "ready", rec.report(t).Status)
400+}
401+
402+func TestBrokerStatusEnsureIncompatibleEndpointFailsWithoutSpawning(t *testing.T) {
403+ layout := emptySandboxRoot(t, "")
404+ deps, rec := testStatusDeps(t)
405+ rec.setProbe(&deps, func(_ int, socketPath string) (brokerStatusReport, error) {
406+ return brokerStatusReport{}, incompatibleError(socketPath, errors.New("foreign endpoint"))
407+ })
408+ err := runBrokerStatus(context.Background(), brokerStatusOptions{offlineRoot: layout.Root, ensure: true, timeout: time.Second}, deps)
409+ require.ErrorIs(t, err, errBrokerIncompatible)
410+ require.Zero(t, rec.spawns.Load())
411+ require.Zero(t, rec.elections.Load())
412+ report := rec.report(t)
413+ require.Equal(t, "offline", report.Status)
414+ require.Equal(t, 3, exitCode(err))
415+}
416+
417+func TestBrokerStatusEnsureBoundedTimeoutReportsOffline(t *testing.T) {
418+ layout := emptySandboxRoot(t, "")
419+ deps, rec := testStatusDeps(t)
420+ err := runBrokerStatus(context.Background(), brokerStatusOptions{offlineRoot: layout.Root, ensure: true, timeout: 50 * time.Millisecond}, deps)
421+ require.ErrorIs(t, err, errBrokerNotReady)
422+ require.Equal(t, 3, exitCode(err))
423+ require.Equal(t, int32(1), rec.spawns.Load())
424+ require.Equal(t, "offline", rec.report(t).Status)
425+}
426+
427+func TestBrokerStatusEnsureSpawnFailureIsReported(t *testing.T) {
428+ layout := emptySandboxRoot(t, "")
429+ spawnErr := errors.New("launcher exec failed")
430+ deps, rec := testStatusDeps(t)
431+ deps.spawn = func(context.Context, string, time.Duration, bool) error {
432+ rec.spawns.Add(1)
433+ return spawnErr
434+ }
435+ err := runBrokerStatus(context.Background(), brokerStatusOptions{offlineRoot: layout.Root, ensure: true, timeout: time.Second}, deps)
436+ require.ErrorIs(t, err, spawnErr)
437+ require.Equal(t, 3, exitCode(err))
438+ require.Equal(t, "offline", rec.report(t).Status)
439+}
440+
441+func TestBrokerStatusEnsureRejectsUnsafeRoot(t *testing.T) {
442+ isolateSandboxEnv(t)
443+ tests := []struct {
444+ name string
445+ root string
446+ }{
447+ {name: "relative root", root: "relative"},
448+ {name: "production overlap", root: brokeripc.SocketDir()},
449+ }
450+ for _, tt := range tests {
451+ t.Run(tt.name, func(t *testing.T) {
452+ deps, rec := testStatusDeps(t)
453+ err := runBrokerStatus(context.Background(), brokerStatusOptions{offlineRoot: tt.root, ensure: true, timeout: time.Second}, deps)
454+ require.Error(t, err)
455+ require.Equal(t, 3, exitCode(err), "a layout failure must still exit 3")
456+ require.Equal(t, "offline", rec.report(t).Status, "a layout failure must still emit one offline report")
457+ require.Zero(t, rec.spawns.Load())
458+ })
459+ }
460+}
461+
462+// TestBrokerStatusEnsureConfigFailureReportsOffline proves a malformed or
463+// missing configuration still leaves the caller with one offline JSON document
464+// and a code-3 failure instead of a bare error.
465+func TestBrokerStatusEnsureConfigFailureReportsOffline(t *testing.T) {
466+ isolateSandboxEnv(t)
467+ root := filepath.Join(shortTempDir(t, "vevm"), "s")
468+ require.NoError(t, os.MkdirAll(root, 0o700))
469+ deps, rec := testStatusDeps(t)
470+ err := runBrokerStatus(context.Background(), brokerStatusOptions{offlineRoot: root, ensure: true, timeout: time.Second}, deps)
471+ require.Error(t, err)
472+ require.Contains(t, err.Error(), brokerconfig.ConfigFileName)
473+ require.Equal(t, 3, exitCode(err))
474+ require.Equal(t, "offline", rec.report(t).Status)
475+ require.Zero(t, rec.spawns.Load())
476+}
477+
478+// TestBrokerStatusEnsureSecuresRootBeforeSpawning proves the ensure path
479+// validates and secures the sandbox root exactly as the launcher does, before
480+// the spawn directory is created or any election lock is taken.
481+func TestBrokerStatusEnsureSecuresRootBeforeSpawning(t *testing.T) {
482+ layout := emptySandboxRoot(t, "")
483+ require.NoError(t, os.Chmod(layout.Root, 0o755))
484+ deps, rec := testStatusDeps(t)
485+ err := runBrokerStatus(context.Background(), brokerStatusOptions{offlineRoot: layout.Root, ensure: true, timeout: time.Second}, deps)
486+ require.Error(t, err)
487+ require.Contains(t, err.Error(), "0755")
488+ require.Equal(t, 3, exitCode(err))
489+ require.Equal(t, "offline", rec.report(t).Status)
490+ require.Zero(t, rec.spawns.Load())
491+ require.Zero(t, rec.elections.Load())
492+ require.NoFileExists(t, filepath.Join(layout.Runtime, brokerconfig.SpawnDirName), "an insecure root must be refused before the spawn directory is created")
493+}
494+
495+// TestBrokerStatusEnsureDeadlineBoundsSpawnWait proves the overall ensure
496+// deadline bounds a launcher spawn that never returns, so a blocking launcher
497+// cannot wedge the status command.
498+func TestBrokerStatusEnsureDeadlineBoundsSpawnWait(t *testing.T) {
499+ layout := emptySandboxRoot(t, "")
500+ deps, rec := testStatusDeps(t)
501+ deps.spawn = func(ctx context.Context, _ string, _ time.Duration, _ bool) error {
502+ <-ctx.Done()
503+ return ctx.Err()
504+ }
505+ start := time.Now()
506+ err := runBrokerStatus(context.Background(), brokerStatusOptions{offlineRoot: layout.Root, ensure: true, timeout: 200 * time.Millisecond}, deps)
507+ require.ErrorIs(t, err, context.DeadlineExceeded)
508+ require.Equal(t, 3, exitCode(err))
509+ require.Less(t, time.Since(start), 5*time.Second, "the overall deadline must bound the launcher spawn wait")
510+ require.Equal(t, "offline", rec.report(t).Status)
511+}
512+
513+// TestBrokerStatusDialOnlyAppliesTimeout proves the dial-only probe is bounded
514+// by --timeout as one absolute bound and is never silently replaced by the
515+// ensure probe budget.
516+func TestBrokerStatusDialOnlyAppliesTimeout(t *testing.T) {
517+ tests := []struct {
518+ name string
519+ timeout time.Duration
520+ }{
521+ {name: "short timeout is the probe bound", timeout: 250 * time.Millisecond},
522+ {name: "default timeout is the probe bound", timeout: brokerStatusDefaultTimeout},
523+ }
524+ for _, tt := range tests {
525+ t.Run(tt.name, func(t *testing.T) {
526+ layout := emptySandboxRoot(t, "")
527+ deps, _ := testStatusDeps(t)
528+ var observed time.Duration
529+ sawDeadline := false
530+ deps.probe = func(ctx context.Context, socketPath string) (brokerStatusReport, error) {
531+ deadline, ok := ctx.Deadline()
532+ sawDeadline = ok
533+ if ok {
534+ observed = time.Until(deadline)
535+ }
536+ return brokerStatusReport{}, absentError(socketPath)
537+ }
538+ require.NoError(t, runBrokerStatus(context.Background(), brokerStatusOptions{offlineRoot: layout.Root, timeout: tt.timeout}, deps))
539+ require.True(t, sawDeadline, "dial-only must bound the probe with the timeout context")
540+ require.InDelta(t, tt.timeout.Seconds(), observed.Seconds(), 0.5)
541+ })
542+ }
543+}
544+
545+// TestBrokerStatusDialOnlyBoundedByTimeout proves a live-but-silent endpoint is
546+// bounded by --timeout rather than the ensure probe budget.
547+func TestBrokerStatusDialOnlyBoundedByTimeout(t *testing.T) {
548+ layout := emptySandboxRoot(t, "")
549+ require.NoError(t, os.MkdirAll(layout.Runtime, 0o700))
550+ _ = bindUnixSocket(t, brokeripc.SocketPath(layout.Runtime))
551+ deps := defaultBrokerStatusDeps()
552+ var out bytes.Buffer
553+ deps.stdout = &out
554+ start := time.Now()
555+ require.NoError(t, runBrokerStatus(context.Background(), brokerStatusOptions{offlineRoot: layout.Root, timeout: 300 * time.Millisecond}, deps))
556+ require.Less(t, time.Since(start), time.Second, "the dial-only probe must be bounded by --timeout, not the ensure budget")
557+ require.Equal(t, "offline", decodeOneReport(t, out.String()).Status)
558+}
559+
560+func TestBrokerStatusEnsureHonorsCancellation(t *testing.T) {
561+ layout := emptySandboxRoot(t, "")
562+ deps, _ := testStatusDeps(t)
563+ ctx, cancel := context.WithCancel(context.Background())
564+ cancel()
565+ err := runBrokerStatus(ctx, brokerStatusOptions{offlineRoot: layout.Root, ensure: true, timeout: time.Second}, deps)
566+ require.ErrorIs(t, err, context.Canceled)
567+}
568+
569+func TestBrokerStatusReportIsBoundedJSON(t *testing.T) {
570+ var out bytes.Buffer
571+ require.NoError(t, writeBrokerStatusReport(&out, brokerStatusReport{
572+ Status: "ready", Endpoint: brokeripc.SocketPath("/tmp/sandbox/runtime"), Epoch: 3, Revision: 9, HostCount: 4,
573+ }))
574+ var fields map[string]json.RawMessage
575+ require.NoError(t, json.Unmarshal([]byte(strings.TrimSpace(out.String())), &fields))
576+ require.Equal(t, []string{"endpoint", "epoch", "host_count", "revision", "status"}, sortedKeys(fields))
577+ require.LessOrEqual(t, out.Len(), brokerStatusMaxReportBytes)
578+}
579+
580+// sortedKeys returns the sorted keys of one JSON object.
581+func sortedKeys(fields map[string]json.RawMessage) []string {
582+ keys := make([]string, 0, len(fields))
583+ for key := range fields {
584+ keys = append(keys, key)
585+ }
586+ slicesSort(keys)
587+ return keys
588+}
589+
590+// slicesSort is a tiny insertion sort so the test file needs no extra import.
591+func slicesSort(values []string) {
592+ for i := 1; i < len(values); i++ {
593+ for j := i; j > 0 && values[j] < values[j-1]; j-- {
594+ values[j], values[j-1] = values[j-1], values[j]
595+ }
596+ }
597+}
598+
599+func TestProbeBrokerStatusClassifiesAbsenceAndIncompatibility(t *testing.T) {
600+ layout := emptySandboxRoot(t, "")
601+ socketPath := brokeripc.SocketPath(layout.Runtime)
602+
603+ t.Run("missing endpoint is absence", func(t *testing.T) {
604+ _, err := probeBrokerStatus(context.Background(), socketPath)
605+ require.ErrorIs(t, err, errBrokerAbsent)
606+ })
607+
608+ t.Run("foreign path is incompatible", func(t *testing.T) {
609+ require.NoError(t, os.MkdirAll(layout.Runtime, 0o700))
610+ require.NoError(t, os.WriteFile(socketPath, []byte("not a socket"), 0o600))
611+ _, err := probeBrokerStatus(context.Background(), socketPath)
612+ require.ErrorIs(t, err, errBrokerIncompatible)
613+ })
614+
615+ t.Run("stale socket is absence", func(t *testing.T) {
616+ require.NoError(t, os.Remove(socketPath))
617+ bindStaleUnixSocket(t, socketPath)
618+ _, err := probeBrokerStatus(context.Background(), socketPath)
619+ require.ErrorIs(t, err, errBrokerAbsent)
620+ })
621+
622+ t.Run("live foreign endpoint is incompatible", func(t *testing.T) {
623+ require.NoError(t, os.Remove(socketPath))
624+ listener := bindUnixSocket(t, socketPath)
625+ go func() {
626+ for {
627+ conn, err := listener.Accept()
628+ if err != nil {
629+ return
630+ }
631+ _ = conn.Close()
632+ }
633+ }()
634+ _, err := probeBrokerStatus(context.Background(), socketPath)
635+ require.ErrorIs(t, err, errBrokerIncompatible)
636+ })
637+}
638+
639+func TestProbeBrokerStatusSilentRegistrationIsBounded(t *testing.T) {
640+ layout := emptySandboxRoot(t, "")
641+ socketPath := brokeripc.SocketPath(layout.Runtime)
642+ require.NoError(t, os.MkdirAll(layout.Runtime, 0o700))
643+ // A listener that never reads or answers: the dial succeeds and the broker
644+ // handshake stalls until the probe budget expires.
645+ _ = bindUnixSocket(t, socketPath)
646+
647+ ctx, cancel := context.WithTimeout(context.Background(), 150*time.Millisecond)
648+ defer cancel()
649+ start := time.Now()
650+ _, err := probeBrokerStatus(ctx, socketPath)
651+ require.Error(t, err)
652+ require.False(t, errors.Is(err, errBrokerAbsent), "a stalled handshake must not be classified as absence")
653+ require.Less(t, time.Since(start), 5*time.Second)
654+}
655+
656+// bindUnixSocket binds one real Unix listener at path.
657+func bindUnixSocket(t *testing.T, path string) net.Listener {
658+ t.Helper()
659+ listener, err := net.Listen("unix", path)
660+ require.NoError(t, err)
661+ t.Cleanup(func() { _ = listener.Close() })
662+ return listener
663+}
664+
665+// bindStaleUnixSocket leaves a socket file behind with no live listener, exactly
666+// like an owner that died without unlinking its endpoint.
667+func bindStaleUnixSocket(t *testing.T, path string) {
668+ t.Helper()
669+ listener, err := net.Listen("unix", path)
670+ require.NoError(t, err)
671+ if unix, ok := listener.(*net.UnixListener); ok {
672+ unix.SetUnlinkOnClose(false)
673+ }
674+ require.NoError(t, listener.Close())
675+ require.FileExists(t, path)
676+}
677+
678+// Ensure the helper fixture contract used elsewhere stays coherent.
679+func TestStatusFixtureDocument(t *testing.T) {
680+ document := sandboxEmptyDocument("1s")
681+ require.Equal(t, brokerconfig.Marker, document["marker"])
682+ require.Equal(t, "1s", document["idleGrace"])
683+ require.Empty(t, sandboxEmptyDocument("")["idleGrace"])
684+}
A internal/app/broker_offline_test.go +715 -0
@@ -0,0 +1,715 @@
1+package app
2+
3+import (
4+ "bytes"
5+ "context"
6+ "encoding/json"
7+ "errors"
8+ "io"
9+ "log/slog"
10+ "os"
11+ "path/filepath"
12+ "sort"
13+ "strings"
14+ "sync"
15+ "sync/atomic"
16+ "testing"
17+ "time"
18+
19+ "github.com/stretchr/testify/require"
20+
21+ "github.com/bnema/vev/internal/adapters/brokerconfig"
22+ "github.com/bnema/vev/internal/adapters/brokeripc"
23+ "github.com/bnema/vev/internal/adapters/daemonmux"
24+ "github.com/bnema/vev/internal/adapters/ipc"
25+ "github.com/bnema/vev/internal/adapters/lifecycle"
26+ "github.com/bnema/vev/internal/domain"
27+ "github.com/bnema/vev/internal/ports"
28+ "github.com/bnema/vev/internal/protocol"
29+)
30+
31+const (
32+ brokerTestEndpoint = "offline@daemon:2222"
33+ brokerTestIdentity = "offline-daemon"
34+ brokerTestWait = 5 * time.Second
35+ brokerTestPollTick = 5 * time.Millisecond
36+ brokerTestIdleAdvance = time.Minute
37+)
38+
39+// brokerTestPolicy is the exact policy provisioned and offered by the fixture.
40+func brokerTestPolicy() ports.BrokerPolicy {
41+ return ports.BrokerPolicy{
42+ ProtocolVersion: protocol.Version,
43+ CatalogSchemaVersion: 3,
44+ EnvironmentPolicy: protocol.EnvironmentPolicyClientOwned,
45+ Transport: "unix-mux",
46+ Trust: "offline-trust",
47+ Launch: "offline-launch",
48+ Isolation: "offline-isolation",
49+ }
50+}
51+
52+// brokerTestRegistration is the exact registration provisioned by the sandbox
53+// config and carried by the request.
54+func brokerTestRegistration() domain.RemoteRegistration {
55+ var registration domain.RemoteRegistration
56+ registration.Endpoint = brokerTestEndpoint
57+ for i := range registration.Incarnation {
58+ registration.Incarnation[i] = byte(i + 1)
59+ }
60+ registration.Generation = 1
61+ return registration
62+}
63+
64+// sandboxRegistrationDocument renders one registration document with the given
65+// route (a Unix path string or an explicit route object).
66+func sandboxRegistrationDocument(route any) map[string]any {
67+ policy := brokerTestPolicy()
68+ return map[string]any{
69+ "marker": brokerconfig.Marker,
70+ "registrations": []any{map[string]any{
71+ "endpoint": brokerTestEndpoint,
72+ "incarnation": "0102030405060708090a0b0c0d0e0f10",
73+ "generation": 1,
74+ "identity": brokerTestIdentity,
75+ "route": route,
76+ "policy": map[string]any{
77+ "protocolVersion": policy.ProtocolVersion,
78+ "catalogSchemaVersion": policy.CatalogSchemaVersion,
79+ "environmentPolicy": "client-owned",
80+ "transport": policy.Transport,
81+ "trust": policy.Trust,
82+ "launch": policy.Launch,
83+ "isolation": policy.Isolation,
84+ },
85+ }},
86+ }
87+}
88+
89+// writeSandboxConfig marshals one document into root/config.json owner-only.
90+func writeSandboxConfig(t *testing.T, root string, document any) {
91+ t.Helper()
92+ raw, err := json.Marshal(document)
93+ require.NoError(t, err)
94+ require.NoError(t, os.WriteFile(filepath.Join(root, brokerconfig.ConfigFileName), raw, 0o600))
95+}
96+
97+// isolateSandboxEnv redirects production XDG roots into private temporary
98+// directories and returns one fresh offline root beside them, so the sandbox
99+// can never touch the real runtime or state paths.
100+func isolateSandboxEnv(t *testing.T) (root, prodRuntime, prodState string) {
101+ t.Helper()
102+ prodRuntime = t.TempDir()
103+ prodState = t.TempDir()
104+ t.Setenv("XDG_RUNTIME_DIR", prodRuntime)
105+ t.Setenv("XDG_STATE_HOME", prodState)
106+ root = filepath.Join(shortTempDir(t, "vevs"), "sandbox")
107+ return root, prodRuntime, prodState
108+}
109+
110+// requireProductionUntouched asserts no production runtime or state entry was
111+// created.
112+func requireProductionUntouched(t *testing.T, prodRuntime, prodState string) {
113+ t.Helper()
114+ for _, dir := range []string{prodRuntime, prodState} {
115+ entries, err := os.ReadDir(dir)
116+ require.NoError(t, err)
117+ require.Empty(t, entries, "production directory %s was touched", dir)
118+ }
119+}
120+
121+// testBrokerServeDeps builds sandbox deps with a deterministic clock and a
122+// discard logger, and reports readiness on the returned channel.
123+func testBrokerServeDeps(clk ports.Clock) (brokerServeDeps, <-chan string) {
124+ deps := defaultBrokerServeDeps()
125+ deps.clock = func() ports.Clock { return clk }
126+ deps.newLogger = func(string) (*slog.Logger, io.Closer, error) {
127+ return slog.New(slog.NewTextHandler(io.Discard, nil)), io.NopCloser(strings.NewReader("")), nil
128+ }
129+ ready := make(chan string, 1)
130+ deps.onReady = func(socketPath string) {
131+ select {
132+ case ready <- socketPath:
133+ default:
134+ }
135+ }
136+ return deps, ready
137+}
138+
139+// runSandbox starts one sandbox serve goroutine and returns its result channel.
140+func runSandbox(ctx context.Context, options brokerServeOptions, deps brokerServeDeps) <-chan error {
141+ done := make(chan error, 1)
142+ go func() { done <- runBrokerServe(ctx, options, deps) }()
143+ return done
144+}
145+
146+// awaitSandboxCancel cancels the sandbox and requires an orderly shutdown.
147+func awaitSandboxCancel(t *testing.T, cancel context.CancelFunc, done <-chan error) {
148+ t.Helper()
149+ cancel()
150+ select {
151+ case err := <-done:
152+ require.NoError(t, err)
153+ case <-time.After(brokerTestWait):
154+ t.Fatal("sandbox did not shut down after cancellation")
155+ }
156+}
157+
158+// fatalListener is a ports.BrokerListener whose Accept always reports one
159+// fixed terminal error, so a test can drive the accept drain without a carriage.
160+type fatalListener struct{ err error }
161+
162+func (l fatalListener) Accept() (ports.BrokerService, error) { return nil, l.err }
163+func (l fatalListener) Close() error { return nil }
164+func (l fatalListener) Addr() string { return "fatal" }
165+
166+// TestDrainBrokerAcceptClassifiesTerminalFailure proves the drain treats a
167+// listener close as an orderly end with no log and no failure signal, while any
168+// other terminal accept failure is logged and reported on the failure channel so
169+// the caller can commit shutdown.
170+func TestDrainBrokerAcceptClassifiesTerminalFailure(t *testing.T) {
171+ tests := []struct {
172+ name string
173+ err error
174+ wantLogged bool
175+ wantFailed bool
176+ }{
177+ {name: "listener close is orderly and quiet", err: brokeripc.ErrListenerClosed},
178+ {name: "unexpected terminal failure is logged and signaled", err: errors.New("carriage accept failed"), wantLogged: true, wantFailed: true},
179+ }
180+ for _, tt := range tests {
181+ t.Run(tt.name, func(t *testing.T) {
182+ var buffer bytes.Buffer
183+ log := slog.New(slog.NewTextHandler(&buffer, nil))
184+ started := make(chan struct{})
185+ drained := make(chan struct{})
186+ failed := make(chan struct{})
187+ go drainBrokerAccept(fatalListener{err: tt.err}, log, started, drained, failed)
188+
189+ select {
190+ case <-started:
191+ case <-time.After(brokerTestWait):
192+ t.Fatal("the drain never started")
193+ }
194+ select {
195+ case <-drained:
196+ case <-time.After(brokerTestWait):
197+ t.Fatal("the drain did not stop on a terminal accept failure")
198+ }
199+
200+ select {
201+ case <-failed:
202+ require.True(t, tt.wantFailed, "an orderly listener close must not signal failure")
203+ default:
204+ require.False(t, tt.wantFailed, "an unexpected terminal failure must signal shutdown")
205+ }
206+
207+ if tt.wantLogged {
208+ require.Contains(t, buffer.String(), "broker_offline_accept_failed")
209+ return
210+ }
211+ require.Empty(t, strings.TrimSpace(buffer.String()), "an orderly listener close must not be logged")
212+ })
213+ }
214+}
215+
216+func TestParseBrokerServeArgs(t *testing.T) {
217+ tests := []struct {
218+ name string
219+ args []string
220+ want brokerServeOptions
221+ wantErr string
222+ }{
223+ {name: "root only", args: []string{"--offline-root", "/srv/sandbox"}, want: brokerServeOptions{offlineRoot: "/srv/sandbox"}},
224+ {
225+ name: "root and grace",
226+ args: []string{"--offline-root", "/srv/sandbox", "--idle-grace", "90s"},
227+ want: brokerServeOptions{offlineRoot: "/srv/sandbox", idleGrace: 90 * time.Second},
228+ },
229+ {name: "missing root", args: nil, wantErr: "requires --offline-root"},
230+ {name: "missing root value", args: []string{"--offline-root"}, wantErr: "requires a path"},
231+ {name: "empty root value", args: []string{"--offline-root", ""}, wantErr: "requires a path"},
232+ {name: "duplicate root", args: []string{"--offline-root", "/a", "--offline-root", "/b"}, wantErr: "duplicate --offline-root"},
233+ {name: "duplicate grace", args: []string{"--offline-root", "/a", "--idle-grace", "1s", "--idle-grace", "2s"}, wantErr: "duplicate --idle-grace"},
234+ {name: "missing grace value", args: []string{"--offline-root", "/a", "--idle-grace"}, wantErr: "requires a duration"},
235+ {name: "invalid duration", args: []string{"--offline-root", "/a", "--idle-grace", "soon"}, wantErr: "not a duration"},
236+ {name: "zero duration", args: []string{"--offline-root", "/a", "--idle-grace", "0s"}, wantErr: "must be positive"},
237+ {name: "negative duration", args: []string{"--offline-root", "/a", "--idle-grace", "-1s"}, wantErr: "must be positive"},
238+ {name: "unknown flag", args: []string{"--offline-root", "/a", "--verbose"}, wantErr: "unknown flag"},
239+ {name: "positional", args: []string{"--offline-root", "/a", "extra"}, wantErr: "positional"},
240+ {name: "equals form is refused", args: []string{"--offline-root=/a"}, wantErr: "unknown flag"},
241+ }
242+ for _, tt := range tests {
243+ t.Run(tt.name, func(t *testing.T) {
244+ command, err := parseBrokerServeArgs(tt.args)
245+ if tt.wantErr != "" {
246+ require.Error(t, err)
247+ require.Contains(t, err.Error(), tt.wantErr)
248+ return
249+ }
250+ require.NoError(t, err)
251+ require.Equal(t, kindBrokerServe, command.kind)
252+ require.Equal(t, tt.want, command.brokerServe)
253+ })
254+ }
255+}
256+
257+func TestBrokerServeIsHiddenFromPublicHelp(t *testing.T) {
258+ require.NotContains(t, usageText, brokerServeCommand)
259+ _, err := parseArgs([]string{brokerServeCommand, "--offline-root", "/srv/sandbox"})
260+ require.NoError(t, err)
261+ _, err = parseArgs([]string{"--help"})
262+ require.NoError(t, err)
263+}
264+
265+func TestBrokerServeRejectsUnsafeRoot(t *testing.T) {
266+ root, _, _ := isolateSandboxEnv(t)
267+ ctx := context.Background()
268+ deps, _ := testBrokerServeDeps(newSandboxClock())
269+
270+ t.Run("relative root", func(t *testing.T) {
271+ err := runBrokerServe(ctx, brokerServeOptions{offlineRoot: "relative"}, deps)
272+ require.Error(t, err)
273+ require.Contains(t, err.Error(), "not absolute")
274+ })
275+ t.Run("unclean root", func(t *testing.T) {
276+ err := runBrokerServe(ctx, brokerServeOptions{offlineRoot: root + "/../sandbox"}, deps)
277+ require.Error(t, err)
278+ require.Contains(t, err.Error(), "not clean")
279+ })
280+ t.Run("production overlap", func(t *testing.T) {
281+ err := runBrokerServe(ctx, brokerServeOptions{offlineRoot: ipc.SocketDir()}, deps)
282+ require.Error(t, err)
283+ require.Contains(t, err.Error(), "overlaps production path")
284+ })
285+ t.Run("symlinked root", func(t *testing.T) {
286+ link := filepath.Join(t.TempDir(), "link")
287+ require.NoError(t, os.Symlink(t.TempDir(), link))
288+ err := runBrokerServe(ctx, brokerServeOptions{offlineRoot: link}, deps)
289+ require.Error(t, err)
290+ require.Contains(t, err.Error(), "symlink")
291+ })
292+ t.Run("missing config", func(t *testing.T) {
293+ err := runBrokerServe(ctx, brokerServeOptions{offlineRoot: root}, deps)
294+ require.Error(t, err)
295+ require.Contains(t, err.Error(), brokerconfig.ConfigFileName)
296+ })
297+}
298+
299+func TestBrokerServeRoundTrip(t *testing.T) {
300+ root, prodRuntime, prodState := isolateSandboxEnv(t)
301+ policy := brokerTestPolicy()
302+ fixture := newBrokerMuxFixture(t, policy)
303+ require.NoError(t, os.MkdirAll(root, 0o700))
304+ writeSandboxConfig(t, root, sandboxRegistrationDocument(fixture.route))
305+
306+ clk := newSandboxClock()
307+ deps, ready := testBrokerServeDeps(clk)
308+ ctx, cancel := context.WithCancel(context.Background())
309+ defer cancel()
310+ done := runSandbox(ctx, brokerServeOptions{offlineRoot: root}, deps)
311+
312+ socketPath := awaitSandboxReady(t, ready)
313+
314+ clientCtx, clientCancel := context.WithTimeout(context.Background(), brokerTestWait)
315+ defer clientCancel()
316+ service, err := brokeripc.Dial(clientCtx, socketPath, brokeripc.Config{})
317+ require.NoError(t, err)
318+ defer func() { _ = service.Close() }()
319+
320+ stream, err := service.OpenStream(clientCtx, ports.BrokerOpenStreamRequest{
321+ Purpose: ports.BrokerStreamControl,
322+ Endpoint: brokerTestEndpoint,
323+ Registration: brokerTestRegistration(),
324+ Policy: policy,
325+ })
326+ require.NoError(t, err)
327+ defer func() { _ = stream.Close() }()
328+
329+ type pingResult struct {
330+ message protocol.ServerMessage
331+ err error
332+ }
333+ result := make(chan pingResult, 1)
334+ go func() {
335+ if sendErr := stream.SendClient(protocol.Ping{}); sendErr != nil {
336+ result <- pingResult{err: sendErr}
337+ return
338+ }
339+ message, recvErr := stream.ReceiveServer()
340+ result <- pingResult{message: message, err: recvErr}
341+ }()
342+ select {
343+ case outcome := <-result:
344+ require.NoError(t, outcome.err)
345+ require.Equal(t, protocol.Pong{}, outcome.message)
346+ case <-time.After(brokerTestWait):
347+ t.Fatal("typed IPC -> pool -> mux round trip timed out")
348+ }
349+ require.Positive(t, fixture.streams.Load())
350+
351+ awaitSandboxCancel(t, cancel, done)
352+ requireSocketRemoved(t, socketPath)
353+ requireProductionUntouched(t, prodRuntime, prodState)
354+}
355+
356+// awaitSandboxReady waits for the sandbox readiness hook and returns the
357+// reported socket path.
358+func awaitSandboxReady(t *testing.T, ready <-chan string) string {
359+ t.Helper()
360+ select {
361+ case socketPath := <-ready:
362+ require.NotEmpty(t, socketPath)
363+ return socketPath
364+ case <-time.After(brokerTestWait):
365+ t.Fatal("sandbox never reported readiness")
366+ return ""
367+ }
368+}
369+
370+// requireSocketRemoved asserts the sandbox socket inode was unlinked.
371+func requireSocketRemoved(t *testing.T, socketPath string) {
372+ t.Helper()
373+ _, err := os.Lstat(socketPath)
374+ require.ErrorIs(t, err, os.ErrNotExist)
375+}
376+
377+// waitForSandboxShutdown advances the manual clock until the sandbox has shut
378+// down, so a real idle grace is never awaited.
379+func waitForSandboxShutdown(t *testing.T, clk *sandboxClock, done <-chan error) {
380+ t.Helper()
381+ deadline := time.After(brokerTestWait)
382+ for {
383+ select {
384+ case err := <-done:
385+ require.NoError(t, err)
386+ return
387+ case <-time.After(brokerTestPollTick):
388+ clk.Advance(brokerTestIdleAdvance)
389+ case <-deadline:
390+ t.Fatal("idle sandbox did not shut down")
391+ }
392+ }
393+}
394+
395+func TestBrokerServeIdleShutdownOnEmptyConfig(t *testing.T) {
396+ root, prodRuntime, prodState := isolateSandboxEnv(t)
397+ clk := newSandboxClock()
398+ require.NoError(t, os.MkdirAll(root, 0o700))
399+ writeSandboxConfig(t, root, map[string]any{"marker": brokerconfig.Marker, "registrations": []any{}})
400+
401+ deps, ready := testBrokerServeDeps(clk)
402+ ctx, cancel := context.WithCancel(context.Background())
403+ defer cancel()
404+ done := runSandbox(ctx, brokerServeOptions{offlineRoot: root}, deps)
405+
406+ socketPath := awaitSandboxReady(t, ready)
407+ waitForSandboxShutdown(t, clk, done)
408+
409+ requireSocketRemoved(t, socketPath)
410+ requireOwnerLockReleased(t, root)
411+ requireProductionUntouched(t, prodRuntime, prodState)
412+}
413+
414+// requireOwnerLockReleased proves the lifetime lock is no longer held.
415+func requireOwnerLockReleased(t *testing.T, root string) {
416+ t.Helper()
417+ owner, err := lifecycle.TryAcquire(filepath.Join(root, brokerconfig.RuntimeDirName))
418+ require.NoError(t, err)
419+ require.NoError(t, owner.Release())
420+}
421+
422+func TestBrokerServeCancelCleanup(t *testing.T) {
423+ root, prodRuntime, prodState := isolateSandboxEnv(t)
424+ clk := newSandboxClock()
425+ require.NoError(t, os.MkdirAll(root, 0o700))
426+ writeSandboxConfig(t, root, map[string]any{"marker": brokerconfig.Marker, "registrations": []any{}})
427+
428+ deps, ready := testBrokerServeDeps(clk)
429+ ctx, cancel := context.WithCancel(context.Background())
430+ defer cancel()
431+ done := runSandbox(ctx, brokerServeOptions{offlineRoot: root}, deps)
432+
433+ socketPath := awaitSandboxReady(t, ready)
434+ awaitSandboxCancel(t, cancel, done)
435+
436+ requireSocketRemoved(t, socketPath)
437+ requireOwnerLockReleased(t, root)
438+ _, err := os.Lstat(filepath.Join(root, brokerconfig.StateDirName, "state.json"))
439+ require.NoError(t, err, "durable sandbox state must remain after shutdown")
440+ requireProductionUntouched(t, prodRuntime, prodState)
441+}
442+
443+// wedgedListener is a ports.BrokerListener whose Accept always reports one
444+// unexpected terminal failure and whose Close records that the sandbox tore it
445+// down, so a test can prove an accept failure does not leave a bound broker.
446+type wedgedListener struct {
447+ err error
448+ addr string
449+ closed atomic.Bool
450+}
451+
452+func (l *wedgedListener) Accept() (ports.BrokerService, error) { return nil, l.err }
453+func (l *wedgedListener) Close() error {
454+ l.closed.Store(true)
455+ return nil
456+}
457+func (l *wedgedListener) Addr() string { return l.addr }
458+
459+// TestBrokerServeShutsDownOnUnexpectedAcceptFailure proves an unexpected
460+// terminal accept failure commits shutdown and drains the listener, instead of
461+// leaving a bound broker whose socket can no longer admit work.
462+func TestBrokerServeShutsDownOnUnexpectedAcceptFailure(t *testing.T) {
463+ root, prodRuntime, prodState := isolateSandboxEnv(t)
464+ clk := newSandboxClock()
465+ require.NoError(t, os.MkdirAll(root, 0o700))
466+ writeSandboxConfig(t, root, map[string]any{"marker": brokerconfig.Marker, "registrations": []any{}})
467+
468+ listener := &wedgedListener{err: errors.New("carriage accept failed"), addr: "wedged"}
469+ deps, ready := testBrokerServeDeps(clk)
470+ deps.listen = func(string, ports.BrokerEpoch, ports.BrokerAuthority, brokeripc.Config) (ports.BrokerListener, error) {
471+ return listener, nil
472+ }
473+ ctx, cancel := context.WithCancel(context.Background())
474+ defer cancel()
475+ done := runSandbox(ctx, brokerServeOptions{offlineRoot: root}, deps)
476+
477+ awaitSandboxReady(t, ready)
478+ select {
479+ case err := <-done:
480+ require.NoError(t, err)
481+ case <-time.After(brokerTestWait):
482+ t.Fatal("the broker wedged on an unexpected accept failure")
483+ }
484+ require.True(t, listener.closed.Load(), "shutdown must close the wedged listener")
485+ requireOwnerLockReleased(t, root)
486+ requireProductionUntouched(t, prodRuntime, prodState)
487+}
488+
489+func TestBrokerServeRefusesDuplicateOwnership(t *testing.T) {
490+ root, _, _ := isolateSandboxEnv(t)
491+ clk := newSandboxClock()
492+ require.NoError(t, os.MkdirAll(root, 0o700))
493+ writeSandboxConfig(t, root, map[string]any{"marker": brokerconfig.Marker, "registrations": []any{}})
494+
495+ deps, ready := testBrokerServeDeps(clk)
496+ ctx, cancel := context.WithCancel(context.Background())
497+ defer cancel()
498+ done := runSandbox(ctx, brokerServeOptions{offlineRoot: root}, deps)
499+ socketPath := awaitSandboxReady(t, ready)
500+
501+ // A second foreground owner fails fast on the lifetime lock.
502+ secondDeps, _ := testBrokerServeDeps(newSandboxClock())
503+ err := runBrokerServe(context.Background(), brokerServeOptions{offlineRoot: root}, secondDeps)
504+ require.ErrorIs(t, err, lifecycle.ErrBusy)
505+
506+ // A duplicate listener on the live socket is refused as well.
507+ duplicate, err := ipc.ListenMux(socketPath)
508+ if duplicate != nil {
509+ _ = duplicate.Close()
510+ }
511+ require.ErrorIs(t, err, ipc.ErrDaemonRunning)
512+
513+ awaitSandboxCancel(t, cancel, done)
514+}
515+
516+// serveTestPongs answers every typed Ping of one accepted daemonmux stream with
517+// exactly one Pong until the stream ends.
518+func serveTestPongs(connection ports.ServerConnection) {
519+ for {
520+ message, err := connection.ReceiveClient()
521+ if err != nil {
522+ return
523+ }
524+ if _, ok := message.(protocol.Ping); !ok {
525+ continue
526+ }
527+ if err := connection.SendServer(protocol.Pong{}); err != nil {
528+ return
529+ }
530+ }
531+}
532+
533+// brokerMuxFixture is a real Unix daemonmux server: one private mux carriage
534+// adopted by a supervisor into an aggregate listener that answers typed Pings.
535+// streams counts admitted typed logical streams so a test can prove the
536+// request actually reached the mux fixture.
537+type brokerMuxFixture struct {
538+ route string
539+ streams atomic.Int64
540+}
541+
542+// newBrokerMuxFixture binds one real private Unix mux carriage and serves typed
543+// Pings on every admitted logical stream.
544+func newBrokerMuxFixture(t *testing.T, policy ports.BrokerPolicy) *brokerMuxFixture {
545+ t.Helper()
546+ routeDir := t.TempDir()
547+ require.NoError(t, os.Chmod(routeDir, 0o700))
548+ route := filepath.Join(routeDir, "mux.sock")
549+ listener, err := ipc.ListenMux(route)
550+ require.NoError(t, err)
551+
552+ aggregate := daemonmux.NewAggregateListener()
553+ var incarnation ports.BrokerDaemonIncarnation
554+ for i := range incarnation {
555+ incarnation[i] = byte(0x10 + i)
556+ }
557+ binding, err := daemonmux.NewServerBinding(ports.BrokerDaemonIdentity(brokerTestIdentity), incarnation, policy)
558+ require.NoError(t, err)
559+ supervisor, err := daemonmux.NewServerSupervisor(aggregate, binding, daemonmux.DefaultMuxCeilings(), 8)
560+ require.NoError(t, err)
561+
562+ fixture := &brokerMuxFixture{route: route}
563+ ctx, cancel := context.WithCancel(context.Background())
564+ t.Cleanup(func() {
565+ cancel()
566+ _ = supervisor.Close()
567+ _ = aggregate.Close()
568+ _ = listener.Close()
569+ })
570+
571+ go func() {
572+ for {
573+ raw, acceptErr := listener.Accept()
574+ if acceptErr != nil {
575+ return
576+ }
577+ go func() { _ = supervisor.Adopt(ctx, raw) }()
578+ }
579+ }()
580+ go func() {
581+ for {
582+ connection, acceptErr := aggregate.Accept()
583+ if acceptErr != nil {
584+ return
585+ }
586+ fixture.streams.Add(1)
587+ go serveTestPongs(connection)
588+ }
589+ }()
590+ return fixture
591+}
592+
593+// sandboxClock is a deterministic ports.Clock/ports.Timer pair: time only moves
594+// when Advance is called, so idle shutdown is exercised without wall time.
595+type sandboxClock struct {
596+ mu sync.Mutex
597+ now time.Time
598+ order uint64
599+ timers map[*sandboxTimer]struct{}
600+}
601+
602+func newSandboxClock() *sandboxClock {
603+ return &sandboxClock{now: time.Unix(0, 0), timers: make(map[*sandboxTimer]struct{})}
604+}
605+
606+func (c *sandboxClock) Now() time.Time {
607+ c.mu.Lock()
608+ defer c.mu.Unlock()
609+ return c.now
610+}
611+
612+func (c *sandboxClock) NewTimer(d time.Duration) ports.Timer {
613+ timer := &sandboxTimer{clock: c, ch: make(chan time.Time, 1)}
614+ c.mu.Lock()
615+ defer c.mu.Unlock()
616+ timer.arm(d)
617+ return timer
618+}
619+
620+// Advance moves time forward and fires every timer that came due.
621+func (c *sandboxClock) Advance(d time.Duration) {
622+ if d < 0 {
623+ panic("sandboxClock.Advance requires a non-negative duration")
624+ }
625+ c.mu.Lock()
626+ defer c.mu.Unlock()
627+ c.now = c.now.Add(d)
628+ due := make([]*sandboxTimer, 0, len(c.timers))
629+ for timer := range c.timers {
630+ if !timer.when.After(c.now) {
631+ due = append(due, timer)
632+ }
633+ }
634+ sort.Slice(due, func(i, j int) bool {
635+ if due[i].when.Equal(due[j].when) {
636+ return due[i].order < due[j].order
637+ }
638+ return due[i].when.Before(due[j].when)
639+ })
640+ for _, timer := range due {
641+ timer.active = false
642+ delete(c.timers, timer)
643+ select {
644+ case timer.ch <- c.now:
645+ default:
646+ }
647+ }
648+}
649+
650+type sandboxTimer struct {
651+ clock *sandboxClock
652+ ch chan time.Time
653+ when time.Time
654+ order uint64
655+ active bool
656+}
657+
658+func (t *sandboxTimer) C() <-chan time.Time { return t.ch }
659+
660+func (t *sandboxTimer) Stop() bool {
661+ t.clock.mu.Lock()
662+ defer t.clock.mu.Unlock()
663+ if !t.active {
664+ return false
665+ }
666+ t.active = false
667+ delete(t.clock.timers, t)
668+ return true
669+}
670+
671+func (t *sandboxTimer) Reset(d time.Duration) bool {
672+ t.clock.mu.Lock()
673+ defer t.clock.mu.Unlock()
674+ wasActive := t.active
675+ if t.active {
676+ t.active = false
677+ delete(t.clock.timers, t)
678+ }
679+ select {
680+ case <-t.ch:
681+ default:
682+ }
683+ t.arm(d)
684+ return wasActive
685+}
686+
687+func (t *sandboxTimer) arm(d time.Duration) {
688+ t.order = t.clock.order
689+ t.clock.order++
690+ t.when = t.clock.now.Add(d)
691+ t.active = true
692+ t.clock.timers[t] = struct{}{}
693+}
694+
695+func TestSandboxClockFiresOnAdvance(t *testing.T) {
696+ clk := newSandboxClock()
697+ timer := clk.NewTimer(time.Minute)
698+ require.True(t, timer.Stop())
699+ timer = clk.NewTimer(time.Minute)
700+ clk.Advance(time.Minute)
701+ select {
702+ case <-timer.C():
703+ case <-time.After(brokerTestWait):
704+ t.Fatal("manual timer did not fire")
705+ }
706+}
707+
708+// Ensure the fixture and registration helpers stay coherent.
709+func TestBrokerSandboxFixtures(t *testing.T) {
710+ require.NoError(t, brokerTestRegistration().Validate())
711+ require.NoError(t, brokerTestPolicy().Validate())
712+ document := sandboxRegistrationDocument("/tmp/mux.sock")
713+ require.Equal(t, brokerconfig.Marker, document["marker"])
714+ require.Len(t, document["registrations"], 1)
715+}
M internal/app/connectivity_matrix.go +42 -0
@@ -216,6 +216,48 @@ var connectivityMatrix = []connectivityEntry{
216 Owner: connectivityLocalOnly,216 Owner: connectivityLocalOnly,
217 Notes: "No connectivity.",217 Notes: "No connectivity.",
218 },218 },
219+ {
220+ Kind: "kindBrokerServe",
221+ Summary: "hidden _broker-serve isolated offline broker sandbox",
222+ Owner: connectivityLocalOnly,
223+ Notes: "Foreground sandbox process over its own private root: no production runtime/state, no daemon dial, no RemoteHostStore, and no session persistence. It composes the broker under a temporary offline config and does not become a client façade or an ordinary command.",
224+ },
225+ {
226+ Kind: "kindBrokerLauncher",
227+ Summary: "hidden _broker-launcher detached offline broker launcher",
228+ Owner: connectivityLocalOnly,
229+ Notes: "Process mechanics only, scoped to one operator-supplied offline root: it starts _broker-serve in a new session and exits, like the daemon launcher, and never dials a daemon, owns the host store, or mutates persistence.",
230+ },
231+ {
232+ Kind: "kindBrokerStatus",
233+ Summary: "hidden _broker-status offline broker probe and connect-or-spawn",
234+ Owner: connectivityLocalOnly,
235+ Notes: "Reports the offline broker's local IPC endpoint and, with --ensure, elects one spawner under a descriptor-backed lock. It dials only the operator-supplied broker sandbox socket (never a daemon or RemoteHostStore) and never becomes a client façade or ordinary command.",
236+ },
237+ {
238+ Kind: "kindBrokerMuxStdio",
239+ Summary: "hidden _broker-mux-stdio remote SSH stdio daemonmux bridge",
240+ Owner: connectivityTransportInfra,
241+ DirectDialDebt: []string{
242+ "ipc.DialMuxContext local daemonmux carriage (broker_offline_mux.go)",
243+ },
244+ Notes: "Remote-side helper over one operator-supplied offline root: it bridges its own stdio to the single provisioned private Unix daemonmux carriage. It never starts a broker, observer, or ordinary daemon, never dials the production daemon socket, and never fabricates a daemon incarnation. Direct-dial debt is recorded here rather than as broker debt because the helper itself stays transport infra.",
245+ },
246+ {
247+ Kind: "kindBrokerMuxQUICBootstrap",
248+ Summary: "hidden _broker-mux-quic-bootstrap remote SSH-side QUIC bootstrap",
249+ Owner: connectivityTransportInfra,
250+ Notes: "Starts one detached _broker-mux-quic-proxy in a new session, forwards its single readiness line, and exits. Its only child is that proxy: it never starts a broker, observer, or ordinary daemon, and one-time QUIC credentials never reach an address, log, or error.",
251+ },
252+ {
253+ Kind: "kindBrokerMuxQUICProxy",
254+ Summary: "hidden _broker-mux-quic-proxy detached remote QUIC daemonmux bridge",
255+ Owner: connectivityTransportInfra,
256+ DirectDialDebt: []string{
257+ "ipc.DialMuxContext local daemonmux carriage (broker_offline_mux.go)",
258+ },
259+ Notes: "Mints one ephemeral authenticated QUIC server, admits exactly one carriage, and bridges it to the single provisioned private Unix daemonmux carriage. It never dials the production daemon socket, never starts a daemon, and never fabricates a daemon incarnation.",
260+ },
219 }261 }
220 262
221 // connectivityOwnerFor returns the matrix owner for a cmdKind identifier.263 // connectivityOwnerFor returns the matrix owner for a cmdKind identifier.
M internal/app/run.go +34 -0
@@ -74,6 +74,12 @@ const (
74 kindWebDaemon74 kindWebDaemon
75 kindWebRenew75 kindWebRenew
76 kindWebServe76 kindWebServe
77+ kindBrokerServe
78+ kindBrokerLauncher
79+ kindBrokerStatus
80+ kindBrokerMuxStdio
81+ kindBrokerMuxQUICBootstrap
82+ kindBrokerMuxQUICProxy
77 kindHelp83 kindHelp
78 kindVersion84 kindVersion
79 )85 )
@@ -92,6 +98,10 @@ type command struct {
92 killAll bool98 killAll bool
93 killDaemon bool99 killDaemon bool
94 cmd cmdInvocation100 cmd cmdInvocation
101+ brokerServe brokerServeOptions
102+ brokerLauncher brokerLauncherOptions
103+ brokerStatus brokerStatusOptions
104+ brokerMux brokerMuxOptions
95 remotePreviewPayload string105 remotePreviewPayload string
96 uiDriver uiDriverOptions106 uiDriver uiDriverOptions
97 uiObserve bool107 uiObserve bool
@@ -230,6 +240,18 @@ parsedUIFlags:
230 return command{kind: kindDaemon}, nil240 return command{kind: kindDaemon}, nil
231 case "--daemon-launcher":241 case "--daemon-launcher":
232 return command{kind: kindDaemonLauncher}, nil242 return command{kind: kindDaemonLauncher}, nil
243+ case brokerServeCommand:
244+ return parseBrokerServeArgs(args[1:])
245+ case brokerLauncherCommand:
246+ return parseBrokerLauncherArgs(args[1:])
247+ case brokerStatusCommand:
248+ return parseBrokerStatusArgs(args[1:])
249+ case brokerMuxStdioCommand:
250+ return parseBrokerMuxArgs(brokerMuxStdioCommand, kindBrokerMuxStdio, args[1:])
251+ case brokerMuxQUICBootstrapCommand:
252+ return parseBrokerMuxArgs(brokerMuxQUICBootstrapCommand, kindBrokerMuxQUICBootstrap, args[1:])
253+ case brokerMuxQUICProxyCommand:
254+ return parseBrokerMuxArgs(brokerMuxQUICProxyCommand, kindBrokerMuxQUICProxy, args[1:])
233 case "_stdio":255 case "_stdio":
234 if len(args) != 1 {256 if len(args) != 1 {
235 return command{}, usagef("`_stdio` does not accept a session name")257 return command{}, usagef("`_stdio` does not accept a session name")
@@ -354,6 +376,18 @@ func dispatch(ctx context.Context, cmd command) error {
354 return runDaemon()376 return runDaemon()
355 case kindDaemonLauncher:377 case kindDaemonLauncher:
356 return runDaemonLauncher()378 return runDaemonLauncher()
379+ case kindBrokerServe:
380+ return runBrokerServeCommand(ctx, cmd.brokerServe)
381+ case kindBrokerLauncher:
382+ return runBrokerLauncherCommand(ctx, cmd.brokerLauncher)
383+ case kindBrokerStatus:
384+ return runBrokerStatusCommand(ctx, cmd.brokerStatus)
385+ case kindBrokerMuxStdio:
386+ return runBrokerMuxStdioCommand(ctx, cmd.brokerMux)
387+ case kindBrokerMuxQUICBootstrap:
388+ return runBrokerMuxQUICBootstrapCommand(ctx, cmd.brokerMux)
389+ case kindBrokerMuxQUICProxy:
390+ return runBrokerMuxQUICProxyCommand(ctx, cmd.brokerMux)
357 case kindStdio:391 case kindStdio:
358 return runStdio(ctx)392 return runStdio(ctx)
359 case kindQUICBootstrap:393 case kindQUICBootstrap:
M internal/app/run_test.go +24 -0
@@ -244,6 +244,30 @@ func TestCatalogueFailureDoesNotListen(t *testing.T) {
244 }244 }
245 }245 }
246 246
247+// TestLifecycleOwnershipAcceptsOwnerOnlyVariantLock is the production-daemon
248+// regression for the relaxed lock-permission rule: a pre-existing lock written
249+// with an owner-only mode other than 0600 must still let the daemon start, while
250+// group/other access stays refused (covered by the lifecycle unit tests).
251+func TestLifecycleOwnershipAcceptsOwnerOnlyVariantLock(t *testing.T) {
252+ runtimeDir := filepath.Join(t.TempDir(), "runtime")
253+ stateDir := filepath.Join(t.TempDir(), "state")
254+ require.NoError(t, safedir.EnsurePrivate(runtimeDir))
255+ require.NoError(t, os.WriteFile(lifecycle.Path(runtimeDir), nil, 0o600))
256+ require.NoError(t, os.Chmod(lifecycle.Path(runtimeDir), 0o700))
257+
258+ started := false
259+ err := runWithLifecycleOwner(context.Background(), runtimeDir, stateDir, func(context.Context) error {
260+ started = true
261+ return nil
262+ })
263+ require.NoError(t, err, "an owner-only variant lock must not block production daemon startup")
264+ require.True(t, started)
265+
266+ owner, err := lifecycle.TryAcquire(runtimeDir)
267+ require.NoError(t, err, "lifecycle ownership must be released after startup")
268+ require.NoError(t, owner.Release())
269+}
270+
247 func TestCatalogueRegistryConstructionPrecedesSocketPublication(t *testing.T) {271 func TestCatalogueRegistryConstructionPrecedesSocketPublication(t *testing.T) {
248 var events []string272 var events []string
249 _, _, err := constructDaemonBeforeSocketPublication(273 _, _, err := constructDaemonBeforeSocketPublication(
M internal/app/spawn_test.go +42 -0
@@ -31,6 +31,27 @@ const (
31 )31 )
32 32
33 func TestMain(m *testing.M) {33 func TestMain(m *testing.M) {
34+ // The hidden offline broker commands are re-executed through this test
35+ // binary, so a broker test can exercise the real dispatch path in a real
36+ // subprocess. The helper env var keeps that interception out of ordinary
37+ // runs.
38+ if len(os.Args) >= 2 && os.Getenv(brokerHelperEnv) == "1" {
39+ switch os.Args[1] {
40+ case brokerServeCommand, brokerLauncherCommand, brokerStatusCommand,
41+ brokerMuxStdioCommand, brokerMuxQUICBootstrapCommand, brokerMuxQUICProxyCommand:
42+ recordBrokerHelperProcess(os.Args[1])
43+ if os.Args[1] == brokerLauncherCommand {
44+ if record := os.Getenv(brokerLauncherBlockRecordEnv); record != "" {
45+ blockBrokerLauncherHelper(record, os.Getenv(brokerLauncherBlockReleaseEnv))
46+ }
47+ }
48+ if err := Run(os.Args[1:]); err != nil {
49+ fmt.Fprintln(os.Stderr, err)
50+ os.Exit(ExitCode(err))
51+ }
52+ os.Exit(0)
53+ }
54+ }
34 if len(os.Args) == 2 {55 if len(os.Args) == 2 {
35 switch os.Args[1] {56 switch os.Args[1] {
36 case "--daemon-launcher":57 case "--daemon-launcher":
@@ -193,6 +214,27 @@ func writeProcessRecord(path string) error {
193 return os.Rename(tmp, path)214 return os.Rename(tmp, path)
194 }215 }
195 216
217+// blockBrokerLauncherHelper is a controlled, blocking launcher: it records its
218+// PID and waits for the release marker (or a bounded fallback) so a test can
219+// hold a launcher open and prove the spawn wait is bounded by its context.
220+func blockBrokerLauncherHelper(record, release string) {
221+ if err := writeProcessRecord(record); err != nil {
222+ os.Exit(2)
223+ }
224+ deadline := time.Now().Add(30 * time.Second)
225+ for time.Now().Before(deadline) {
226+ _, statErr := os.Stat(release)
227+ if statErr == nil {
228+ break
229+ }
230+ if !errors.Is(statErr, os.ErrNotExist) {
231+ os.Exit(2)
232+ }
233+ time.Sleep(10 * time.Millisecond)
234+ }
235+ os.Exit(0)
236+}
237+
196 func terminateProcessFromFile(path string) error {238 func terminateProcessFromFile(path string) error {
197 data, err := os.ReadFile(path)239 data, err := os.ReadFile(path)
198 if errors.Is(err, os.ErrNotExist) {240 if errors.Is(err, os.ErrNotExist) {
M internal/ports/broker.go +18 -0
@@ -680,6 +680,24 @@ type BrokerEndpointConnector interface {
680 Connect(ctx context.Context, endpoint BrokerResolvedEndpoint) (BrokerPhysicalConnection, error)680 Connect(ctx context.Context, endpoint BrokerResolvedEndpoint) (BrokerPhysicalConnection, error)
681 }681 }
682 682
683+// BrokerAuthority admits one accepted client connection to the broker core
684+// and returns the BrokerService scoped to exactly that connection. It is the
685+// admission seam between the broker listener adapter and the broker use case:
686+// the listener owns the carriage and calls AdmitClient once per accepted
687+// connection, the implementation assigns the connection identity, and the
688+// returned service owns that connection until Close. The adapter never
689+// constructs a broker itself.
690+//
691+// ctx bounds admission only. It carries the listener's handshake deadline and
692+// is canceled as soon as admission returns or when the listener closes, so an
693+// implementation must not retain it or start work that outlives admission
694+// from it; honoring that cancellation is what lets a listener drop a client
695+// parked in admission immediately. The admitted service gets its own
696+// connection-lived context and must not derive from this one.
697+type BrokerAuthority interface {
698+ AdmitClient(ctx context.Context) (BrokerService, error)
699+}
700+
683 // BrokerListener accepts client connections to the per-user broker701 // BrokerListener accepts client connections to the per-user broker
684 // endpoint. Each Accept returns a BrokerService bound to exactly one702 // endpoint. Each Accept returns a BrokerService bound to exactly one
685 // accepted client connection: the implementation assigns that703 // accepted client connection: the implementation assigns that
M internal/ports/broker_test.go +1 -0
@@ -535,6 +535,7 @@ func TestBrokerInterfacesArePorts(t *testing.T) {
535 "BrokerHostProbe": reflect.TypeOf((*BrokerHostProbe)(nil)).Elem(),535 "BrokerHostProbe": reflect.TypeOf((*BrokerHostProbe)(nil)).Elem(),
536 "BrokerPhysicalConnection": reflect.TypeOf((*BrokerPhysicalConnection)(nil)).Elem(),536 "BrokerPhysicalConnection": reflect.TypeOf((*BrokerPhysicalConnection)(nil)).Elem(),
537 "BrokerEndpointConnector": reflect.TypeOf((*BrokerEndpointConnector)(nil)).Elem(),537 "BrokerEndpointConnector": reflect.TypeOf((*BrokerEndpointConnector)(nil)).Elem(),
538+ "BrokerAuthority": reflect.TypeOf((*BrokerAuthority)(nil)).Elem(),
538 "BrokerListener": reflect.TypeOf((*BrokerListener)(nil)).Elem(),539 "BrokerListener": reflect.TypeOf((*BrokerListener)(nil)).Elem(),
539 "BrokerSubscription": reflect.TypeOf((*BrokerSubscription)(nil)).Elem(),540 "BrokerSubscription": reflect.TypeOf((*BrokerSubscription)(nil)).Elem(),
540 } {541 } {
M internal/ports/mocks/broker_mock.go +89 -0
@@ -1348,6 +1348,95 @@ func (_c *MockBrokerEndpointConnector_Connect_Call) RunAndReturn(run func(ctx co
1348 return _c1348 return _c
1349 }1349 }
1350 1350
1351+// NewMockBrokerAuthority creates a new instance of MockBrokerAuthority. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
1352+// The first argument is typically a *testing.T value.
1353+func NewMockBrokerAuthority(t interface {
1354+ mock.TestingT
1355+ Cleanup(func())
1356+}) *MockBrokerAuthority {
1357+ mock := &MockBrokerAuthority{}
1358+ mock.Mock.Test(t)
1359+
1360+ t.Cleanup(func() { mock.AssertExpectations(t) })
1361+
1362+ return mock
1363+}
1364+
1365+// MockBrokerAuthority is an autogenerated mock type for the BrokerAuthority type
1366+type MockBrokerAuthority struct {
1367+ mock.Mock
1368+}
1369+
1370+type MockBrokerAuthority_Expecter struct {
1371+ mock *mock.Mock
1372+}
1373+
1374+func (_m *MockBrokerAuthority) EXPECT() *MockBrokerAuthority_Expecter {
1375+ return &MockBrokerAuthority_Expecter{mock: &_m.Mock}
1376+}
1377+
1378+// AdmitClient provides a mock function for the type MockBrokerAuthority
1379+func (_mock *MockBrokerAuthority) AdmitClient(ctx context.Context) (ports.BrokerService, error) {
1380+ ret := _mock.Called(ctx)
1381+
1382+ if len(ret) == 0 {
1383+ panic("no return value specified for AdmitClient")
1384+ }
1385+
1386+ var r0 ports.BrokerService
1387+ var r1 error
1388+ if returnFunc, ok := ret.Get(0).(func(context.Context) (ports.BrokerService, error)); ok {
1389+ return returnFunc(ctx)
1390+ }
1391+ if returnFunc, ok := ret.Get(0).(func(context.Context) ports.BrokerService); ok {
1392+ r0 = returnFunc(ctx)
1393+ } else {
1394+ if ret.Get(0) != nil {
1395+ r0 = ret.Get(0).(ports.BrokerService)
1396+ }
1397+ }
1398+ if returnFunc, ok := ret.Get(1).(func(context.Context) error); ok {
1399+ r1 = returnFunc(ctx)
1400+ } else {
1401+ r1 = ret.Error(1)
1402+ }
1403+ return r0, r1
1404+}
1405+
1406+// MockBrokerAuthority_AdmitClient_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'AdmitClient'
1407+type MockBrokerAuthority_AdmitClient_Call struct {
1408+ *mock.Call
1409+}
1410+
1411+// AdmitClient is a helper method to define mock.On call
1412+// - ctx context.Context
1413+func (_e *MockBrokerAuthority_Expecter) AdmitClient(ctx any) *MockBrokerAuthority_AdmitClient_Call {
1414+ return &MockBrokerAuthority_AdmitClient_Call{Call: _e.mock.On("AdmitClient", ctx)}
1415+}
1416+
1417+func (_c *MockBrokerAuthority_AdmitClient_Call) Run(run func(ctx context.Context)) *MockBrokerAuthority_AdmitClient_Call {
1418+ _c.Call.Run(func(args mock.Arguments) {
1419+ var arg0 context.Context
1420+ if args[0] != nil {
1421+ arg0 = args[0].(context.Context)
1422+ }
1423+ run(
1424+ arg0,
1425+ )
1426+ })
1427+ return _c
1428+}
1429+
1430+func (_c *MockBrokerAuthority_AdmitClient_Call) Return(brokerService ports.BrokerService, err error) *MockBrokerAuthority_AdmitClient_Call {
1431+ _c.Call.Return(brokerService, err)
1432+ return _c
1433+}
1434+
1435+func (_c *MockBrokerAuthority_AdmitClient_Call) RunAndReturn(run func(ctx context.Context) (ports.BrokerService, error)) *MockBrokerAuthority_AdmitClient_Call {
1436+ _c.Call.Return(run)
1437+ return _c
1438+}
1439+
1351 // NewMockBrokerListener creates a new instance of MockBrokerListener. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.1440 // NewMockBrokerListener creates a new instance of MockBrokerListener. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
1352 // The first argument is typically a *testing.T value.1441 // The first argument is typically a *testing.T value.
1353 func NewMockBrokerListener(t interface {1442 func NewMockBrokerListener(t interface {
M internal/usecase/broker/registry.go +42 -2
@@ -37,6 +37,17 @@ const (
37 runStopped37 runStopped
38 )38 )
39 39
40+// RegistryConfig selects how a Registry runs. The zero value observes hosts.
41+type RegistryConfig struct {
42+ // ObservationDisabled runs the registry as a read-only snapshot owner: it
43+ // restores and publishes durable state, owns and drains its writer, and
44+ // honors the Run lifecycle, but issues no probes, arms no timers, and never
45+ // reconciles. The probe dependency is unused and may be nil. Membership is
46+ // expected to be immutable for the run; a caller that needs observation
47+ // keeps the zero-value config.
48+ ObservationDisabled bool
49+}
50+
40 // Registry owns configured hosts and their immutable observation projection.51 // Registry owns configured hosts and their immutable observation projection.
41 type Registry struct {52 type Registry struct {
42 epoch ports.BrokerEpoch53 epoch ports.BrokerEpoch
@@ -44,6 +55,9 @@ type Registry struct {
44 clock ports.Clock55 clock ports.Clock
45 log *slog.Logger56 log *slog.Logger
46 store *snapshotWriter57 store *snapshotWriter
58+ // observationDisabled is fixed at construction: a disabled registry never
59+ // observes, so it has no probe, timer, or reconcile work to do.
60+ observationDisabled bool
47 61
48 // hostStore and authority are the synchronous membership owner. Only this62 // hostStore and authority are the synchronous membership owner. Only this
49 // registry may mutate the store while it is live; the caller owns Close.63 // registry may mutate the store while it is live; the caller owns Close.
@@ -94,7 +108,18 @@ type Registry struct {
94 // outside Registry.ReplaceHosts. The caller closes the store after the registry108 // outside Registry.ReplaceHosts. The caller closes the store after the registry
95 // has stopped and flushed its advisory writer.109 // has stopped and flushed its advisory writer.
96 func NewRegistry(epoch ports.BrokerEpoch, store ports.BrokerHostStore, probe ports.BrokerHostProbe, clock ports.Clock, log *slog.Logger) (*Registry, error) {110 func NewRegistry(epoch ports.BrokerEpoch, store ports.BrokerHostStore, probe ports.BrokerHostProbe, clock ports.Clock, log *slog.Logger) (*Registry, error) {
97- if epoch == 0 || nilDependency(store) || nilDependency(probe) || nilDependency(clock) {111+ return NewRegistryWithConfig(epoch, store, probe, clock, log, RegistryConfig{})
112+}
113+
114+// NewRegistryWithConfig restores a validated durable snapshot under a fresh
115+// broker epoch and selects the run mode from cfg. Observation-disabled mode
116+// tolerates a nil probe because such a registry never probes; every other
117+// dependency is required exactly as NewRegistry requires it.
118+func NewRegistryWithConfig(epoch ports.BrokerEpoch, store ports.BrokerHostStore, probe ports.BrokerHostProbe, clock ports.Clock, log *slog.Logger, cfg RegistryConfig) (*Registry, error) {
119+ if epoch == 0 || nilDependency(store) || nilDependency(clock) {
120+ return nil, errors.New("broker: invalid registry dependencies")
121+ }
122+ if !cfg.ObservationDisabled && nilDependency(probe) {
98 return nil, errors.New("broker: invalid registry dependencies")123 return nil, errors.New("broker: invalid registry dependencies")
99 }124 }
100 if log == nil {125 if log == nil {
@@ -102,7 +127,8 @@ func NewRegistry(epoch ports.BrokerEpoch, store ports.BrokerHostStore, probe por
102 }127 }
103 r := &Registry{128 r := &Registry{
104 epoch: epoch, probe: probe, clock: clock, log: log,129 epoch: epoch, probe: probe, clock: clock, log: log,
105- hosts: make(map[string]ports.RemoteHostSnapshot), inflight: make(map[string]*probeAttempt),130+ observationDisabled: cfg.ObservationDisabled,
131+ hosts: make(map[string]ports.RemoteHostSnapshot), inflight: make(map[string]*probeAttempt),
106 pending: make(map[string]bool), tombstones: make(map[string]ports.BrokerHostTombstone),132 pending: make(map[string]bool), tombstones: make(map[string]ports.BrokerHostTombstone),
107 subs: make(map[*subscription]struct{}), wake: make(chan struct{}, 1),133 subs: make(map[*subscription]struct{}), wake: make(chan struct{}, 1),
108 freshFor: defaultFreshFor, retryBase: defaultRetryBase, retryMax: defaultRetryLimit,134 freshFor: defaultFreshFor, retryBase: defaultRetryBase, retryMax: defaultRetryLimit,
@@ -350,7 +376,12 @@ func (r *Registry) cancelAllInflightLocked() {
350 }376 }
351 377
352 // RequestProbe is non-blocking and coalesces concurrent demand per endpoint.378 // RequestProbe is non-blocking and coalesces concurrent demand per endpoint.
379+// A disabled registry never observes, so the demand is dropped without
380+// scheduling anything.
353 func (r *Registry) RequestProbe(endpoint string) {381 func (r *Registry) RequestProbe(endpoint string) {
382+ if r.observationDisabled {
383+ return
384+ }
354 r.mu.Lock()385 r.mu.Lock()
355 if _, ok := r.hosts[endpoint]; ok {386 if _, ok := r.hosts[endpoint]; ok {
356 r.pending[endpoint] = true387 r.pending[endpoint] = true
@@ -371,12 +402,21 @@ func (r *Registry) hint() {
371 // concurrent or restarted caller can never duplicate scheduling. On return it402 // concurrent or restarted caller can never duplicate scheduling. On return it
372 // clears in-flight observation state, publishes the settled projection, and403 // clears in-flight observation state, publishes the settled projection, and
373 // flushes durable persistence before any further write is refused.404 // flushes durable persistence before any further write is refused.
405+//
406+// An observation-disabled registry runs the same lifecycle and drain but never
407+// schedules observation: it arms no timer, starts no probe, and drops every
408+// reconcile. It still owns and drains its durable writer, so a shutdown never
409+// returns before the newest staged publication has reached the store.
374 func (r *Registry) Run(ctx context.Context) {410 func (r *Registry) Run(ctx context.Context) {
375 if !r.running.CompareAndSwap(runIdle, runActive) {411 if !r.running.CompareAndSwap(runIdle, runActive) {
376 r.log.Warn("broker: registry Run refused; single-run already started or finished")412 r.log.Warn("broker: registry Run refused; single-run already started or finished")
377 return413 return
378 }414 }
379 defer r.settle()415 defer r.settle()
416+ if r.observationDisabled {
417+ <-ctx.Done()
418+ return
419+ }
380 timer := r.clock.NewTimer(0)420 timer := r.clock.NewTimer(0)
381 defer timer.Stop()421 defer timer.Stop()
382 results := make(chan probeResult, ports.BrokerMaxHosts)422 results := make(chan probeResult, ports.BrokerMaxHosts)
A internal/usecase/broker/service.go +301 -0
@@ -0,0 +1,301 @@
1+package broker
2+
3+import (
4+ "context"
5+ "errors"
6+ "sync"
7+
8+ "github.com/bnema/vev/internal/ports"
9+)
10+
11+// Offline broker composition (Plan 001 P3.4 slice B).
12+//
13+// Authority admits one accepted client connection to the broker core and
14+// returns the connection-scoped ports.BrokerService for exactly that
15+// connection. It is the structural implementation of ports.BrokerAuthority,
16+// the admission seam the broker IPC listener (internal/adapters/brokeripc)
17+// consumes and which this use case must not import: admission obtains one
18+// supervisor client lease and one pool connection
19+// identity, and rolls both back on any failure so a refused admission never
20+// pins the broker open or leaks a client slot. The admission context bounds
21+// admission only and is never retained; the admitted service owns a
22+// connection-lived context derived from the supervisor root.
23+//
24+// This composition is deliberately offline and immutable: its registry runs
25+// observation-disabled, so the returned service publishes and subscribes to
26+// durable snapshots but never probes or reconciles, and host membership is
27+// refused rather than mutated. Live membership mutation is composed in a later
28+// slice.
29+
30+// ErrOfflineMembership is the typed refusal returned by AddHost and RemoveHost
31+// on an offline broker connection: membership is immutable for the
32+// connection's lifetime and the refusal never touches durable authority.
33+// Callers classify it with errors.Is instead of matching message text.
34+var ErrOfflineMembership = errors.New("broker: offline membership is immutable")
35+
36+// offlineMembershipRefusal is the typed, presentation-safe refusal. It is a
37+// ports.BrokerError so the wire layer classifies it without message matching,
38+// and it carries ErrOfflineMembership for errors.Is.
39+func offlineMembershipRefusal() error {
40+ return ports.BrokerError{Code: ports.BrokerErrorUnavailable, Text: ErrOfflineMembership.Error(), Cause: ErrOfflineMembership}
41+}
42+
43+// Authority admits one accepted client connection to the broker core. It owns
44+// no lifetime of its own: the supervisor owns the broker lifetime, the registry
45+// owns snapshots, and the pool owns transports. An admitted Service is bound to
46+// exactly one pool client identity and one broker epoch.
47+//
48+// It is the structural implementation of ports.BrokerAuthority, the admission
49+// seam the broker IPC listener consumes; this use case never imports that
50+// adapter.
51+type Authority struct {
52+ epoch ports.BrokerEpoch
53+ registry *Registry
54+ pool *Pool
55+ supervisor *Supervisor
56+}
57+
58+var _ ports.BrokerAuthority = (*Authority)(nil)
59+
60+// NewAuthority composes one connection-scoped broker authority over an existing
61+// registry, pool, and supervisor. All three must be live and share epoch. The
62+// registry must be observation-disabled: this authority's sessions expose
63+// immutable offline membership, so an observing registry would probe hosts the
64+// service refuses to manage.
65+//
66+// The caller owns the dependencies' lifecycles. In particular it must start the
67+// registry's single Registry.Run to own and drain the durable writer, and must
68+// cancel that run context and join Run before the store is closed. NewAuthority
69+// neither starts nor settles Run: serving connections without a live Run leaks
70+// the writer and never flushes the newest staged publication, and a registry
71+// that was never run leaves its durable writer undrained.
72+func NewAuthority(epoch ports.BrokerEpoch, registry *Registry, pool *Pool, supervisor *Supervisor) (*Authority, error) {
73+ if epoch == 0 || nilDependency(registry) || nilDependency(pool) || nilDependency(supervisor) {
74+ return nil, errors.New("broker: invalid authority dependencies")
75+ }
76+ if registry.epoch != epoch || pool.epoch != epoch {
77+ return nil, errors.New("broker: authority epoch mismatch")
78+ }
79+ if !registry.observationDisabled {
80+ return nil, errors.New("broker: authority requires an observation-disabled registry")
81+ }
82+ return &Authority{epoch: epoch, registry: registry, pool: pool, supervisor: supervisor}, nil
83+}
84+
85+// AdmitClient admits exactly one accepted client connection. The supervisor
86+// client lease and the pool connection identity are acquired together; if
87+// either fails, the other is released before returning, so admission either
88+// commits both or neither. ctx bounds admission only and is never retained: the
89+// returned service derives its own connection-lived context from the supervisor
90+// root, so a caller's setup deadline cannot disturb an admitted connection.
91+func (a *Authority) AdmitClient(ctx context.Context) (ports.BrokerService, error) {
92+ if err := ctx.Err(); err != nil {
93+ return nil, err
94+ }
95+ lease, err := a.supervisor.AdmitClient()
96+ if err != nil {
97+ return nil, err
98+ }
99+ id, err := a.pool.RegisterClient()
100+ if err != nil {
101+ lease.Release()
102+ return nil, err
103+ }
104+ if err := ctx.Err(); err != nil {
105+ a.pool.CloseClient(id)
106+ lease.Release()
107+ return nil, err
108+ }
109+ return newService(a.epoch, id, a.registry, a.pool, a.supervisor, lease), nil
110+}
111+
112+// Service is one admitted client connection: the core's service for exactly one
113+// pool connection identity and broker epoch. It implements ports.BrokerService.
114+type Service struct {
115+ epoch ports.BrokerEpoch
116+ id ports.BrokerConnectionID
117+ registry *Registry
118+ pool *Pool
119+ supervisor *Supervisor
120+ lease *Lease
121+
122+ // ctx owns everything this connection started: in-flight setup opens and
123+ // live streams. It is derived from the supervisor root, never from the
124+ // admission context, so broker shutdown and Close both abort owned work.
125+ ctx context.Context
126+ cancel context.CancelFunc
127+
128+ mu sync.Mutex
129+ closed bool
130+ subs map[*serviceSubscription]struct{}
131+ // wg counts in-flight OpenStream calls so Close drains them, and their
132+ // operation leases, before it releases the connection resources.
133+ wg sync.WaitGroup
134+
135+ closeOnce sync.Once
136+}
137+
138+var _ ports.BrokerService = (*Service)(nil)
139+
140+func newService(epoch ports.BrokerEpoch, id ports.BrokerConnectionID, registry *Registry, pool *Pool, supervisor *Supervisor, lease *Lease) *Service {
141+ ctx, cancel := context.WithCancel(supervisor.RootContext())
142+ return &Service{
143+ epoch: epoch, id: id, registry: registry, pool: pool, supervisor: supervisor, lease: lease,
144+ ctx: ctx, cancel: cancel, subs: make(map[*serviceSubscription]struct{}),
145+ }
146+}
147+
148+// ConnectionID returns the pool identity assigned at admission. Clients carry
149+// it on stream operations so a stale request can be fenced.
150+func (s *Service) ConnectionID() ports.BrokerConnectionID { return s.id }
151+
152+// Snapshot delegates to the registry's current immutable publication.
153+func (s *Service) Snapshot() ports.BrokerSnapshot { return s.registry.Snapshot() }
154+
155+// Subscribe delegates to the registry and tracks the returned subscription so
156+// Close releases every subscription this connection still owns. A closed
157+// connection refuses new subscriptions.
158+func (s *Service) Subscribe() (ports.BrokerSubscription, error) {
159+ s.mu.Lock()
160+ if s.closed {
161+ s.mu.Unlock()
162+ return nil, ports.BrokerAdmissionClosed
163+ }
164+ sub := &serviceSubscription{service: s, inner: s.registry.Subscribe()}
165+ s.subs[sub] = struct{}{}
166+ s.mu.Unlock()
167+ return sub, nil
168+}
169+
170+// OpenStream opens one logical stream through the pool on behalf of the user.
171+// The request is fenced to this connection's exact epoch and identity before
172+// the pool sees it, and one operation lease pins the broker only while setup is
173+// in flight: a stream that opens successfully outlives its setup lease, and
174+// Close or broker shutdown still aborts anything that has not finished.
175+func (s *Service) OpenStream(ctx context.Context, request ports.BrokerOpenStreamRequest) (ports.BrokerLogicalConnection, error) {
176+ if err := s.scope(&request); err != nil {
177+ return nil, err
178+ }
179+ s.mu.Lock()
180+ if s.closed {
181+ s.mu.Unlock()
182+ return nil, ports.BrokerAdmissionClosed
183+ }
184+ s.wg.Add(1)
185+ s.mu.Unlock()
186+ defer s.wg.Done()
187+
188+ operation, err := s.supervisor.AdmitOperation()
189+ if err != nil {
190+ return nil, err
191+ }
192+ defer operation.Release()
193+
194+ openCtx, release := s.openContext(ctx)
195+ stream, err := s.pool.OpenStream(openCtx, request)
196+ if err != nil {
197+ release()
198+ return nil, err
199+ }
200+ // Detach the connection-scoped context link once the pool reports the
201+ // stream terminal, so a long-lived connection never accumulates one link
202+ // per opened stream. Nothing here ends the stream: setup completion and the
203+ // operation lease's release both happen while it keeps running.
204+ go func() {
205+ <-stream.Done()
206+ release()
207+ }()
208+ return stream, nil
209+}
210+
211+// CloseStream retires one stream of this connection. A foreign connection
212+// identity is refused as stale and never retires another connection's stream.
213+func (s *Service) CloseStream(connection ports.BrokerConnectionID, stream ports.BrokerStreamID) error {
214+ if !connection.IsZero() && connection != s.id {
215+ return ports.BrokerAdmissionStale
216+ }
217+ return s.pool.CloseStream(s.id, stream)
218+}
219+
220+// AddHost refuses host admission: this connection has immutable offline
221+// membership, so no mutation is attempted.
222+func (s *Service) AddHost(context.Context, string) error { return offlineMembershipRefusal() }
223+
224+// RemoveHost refuses host removal without mutating authority. The result is the
225+// explicit false/refusal pair.
226+func (s *Service) RemoveHost(context.Context, string) (bool, error) {
227+ return false, offlineMembershipRefusal()
228+}
229+
230+// RequestReconcile is deliberately a no-op: an offline connection has no
231+// observation to reconcile, so the hint starts no probe, timer, or reconcile.
232+func (s *Service) RequestReconcile(string) {}
233+
234+// Close cancels and drains every owned resource exactly once: in-flight setup
235+// opens and live streams stop, subscriptions close, the pool client is retired,
236+// and the supervisor client lease is released. It is idempotent and
237+// concurrent-safe; every caller observes the same terminal state.
238+func (s *Service) Close() error {
239+ s.closeOnce.Do(func() {
240+ s.mu.Lock()
241+ s.closed = true
242+ subs := make([]*serviceSubscription, 0, len(s.subs))
243+ for sub := range s.subs {
244+ subs = append(subs, sub)
245+ }
246+ s.mu.Unlock()
247+
248+ s.cancel()
249+ s.wg.Wait()
250+ for _, sub := range subs {
251+ sub.Close()
252+ }
253+ s.pool.CloseClient(s.id)
254+ s.lease.Release()
255+ })
256+ return nil
257+}
258+
259+// scope fences one request to this connection's exact epoch and identity,
260+// filling an absent identity with this connection's own. A foreign epoch or
261+// connection is refused before it can steer another connection's work.
262+func (s *Service) scope(request *ports.BrokerOpenStreamRequest) error {
263+ if request.Epoch != 0 && request.Epoch != s.epoch {
264+ return ports.BrokerError{Code: ports.BrokerErrorStaleEpoch}
265+ }
266+ if !request.Connection.IsZero() && request.Connection != s.id {
267+ return ports.BrokerAdmissionStale
268+ }
269+ request.Epoch = s.epoch
270+ request.Connection = s.id
271+ return nil
272+}
273+
274+// openContext derives the context that owns one stream: the caller's context
275+// bounds the request, and the connection context bounds the service, so the
276+// stream and its setup end when either does. The returned release detaches the
277+// connection link once the stream has settled.
278+func (s *Service) openContext(ctx context.Context) (context.Context, func()) {
279+ openCtx, cancel := context.WithCancel(ctx)
280+ stop := context.AfterFunc(s.ctx, cancel)
281+ return openCtx, func() { stop(); cancel() }
282+}
283+
284+// serviceSubscription is one connection-owned subscription. Closing it releases
285+// the registry subscription and forgets it on the owning service.
286+type serviceSubscription struct {
287+ service *Service
288+ inner ports.BrokerSubscription
289+ once sync.Once
290+}
291+
292+func (s *serviceSubscription) Changed() <-chan struct{} { return s.inner.Changed() }
293+
294+func (s *serviceSubscription) Close() {
295+ s.once.Do(func() {
296+ s.inner.Close()
297+ s.service.mu.Lock()
298+ delete(s.service.subs, s)
299+ s.service.mu.Unlock()
300+ })
301+}
A internal/usecase/broker/service_test.go +508 -0
@@ -0,0 +1,508 @@
1+package broker
2+
3+import (
4+ "context"
5+ "fmt"
6+ "sync"
7+ "testing"
8+ "time"
9+
10+ "github.com/bnema/vev/internal/domain"
11+ "github.com/bnema/vev/internal/ports"
12+ "github.com/stretchr/testify/require"
13+)
14+
15+// newTestAuthority composes one offline authority over an observation-disabled
16+// registry, a bounded pool, and a supervisor on a long grace. Every dependency
17+// is drained at test end so no run goroutine leaks between tests: the registry
18+// run is canceled and joined (which flushes and stops its durable writer), the
19+// pool is closed, and the supervisor is closed.
20+func newTestAuthority(t testing.TB, epoch ports.BrokerEpoch, probe ports.BrokerHostProbe, connector poolConnector) (*Authority, *Registry, *Pool, *Supervisor, *manualClock, *testStore) {
21+ t.Helper()
22+ clock := newManualClock(time.Unix(0, 0))
23+ store := newTestStore()
24+ registry, err := NewRegistryWithConfig(epoch, store, probe, clock, nil, RegistryConfig{ObservationDisabled: true})
25+ require.NoError(t, err)
26+ resolver := poolResolver(func(_ context.Context, r ports.BrokerOpenStreamRequest) (ports.BrokerResolvedEndpoint, error) {
27+ return ports.BrokerResolvedEndpoint{Identity: "canonical", Policy: r.Policy, Address: "fake"}, nil
28+ })
29+ pool, err := NewPool(epoch, resolver, connector, clock, PoolLimits{Physical: 4, Clients: 8, Streams: 16, StreamsPerClient: 16, Idle: time.Minute})
30+ require.NoError(t, err)
31+ t.Cleanup(func() { require.NoError(t, pool.Close()) })
32+ supervisor, _ := newTestSupervisor(t, time.Hour)
33+ authority, err := NewAuthority(epoch, registry, pool, supervisor)
34+ require.NoError(t, err)
35+ // The caller owns Registry.Run (see NewAuthority): start the single run and
36+ // cancel it at cleanup, so the registry's durable writer is drained and its
37+ // goroutine never survives the test.
38+ registryCtx, stopRegistry := context.WithCancel(context.Background())
39+ registryDone := make(chan struct{})
40+ go func() { defer close(registryDone); registry.Run(registryCtx) }()
41+ t.Cleanup(func() { stopRegistry(); <-registryDone })
42+ return authority, registry, pool, supervisor, clock, store
43+}
44+
45+// immediateConnector returns one healthy physical connection for any endpoint.
46+func immediateConnector(_ context.Context, e ports.BrokerResolvedEndpoint) (ports.BrokerPhysicalConnection, error) {
47+ return &fakePhysical{endpoint: e, done: make(chan struct{})}, nil
48+}
49+
50+func poolClientCount(p *Pool) int {
51+ p.mu.Lock()
52+ defer p.mu.Unlock()
53+ return len(p.clients)
54+}
55+
56+func registrySubscriptionCount(r *Registry) int {
57+ r.mu.Lock()
58+ defer r.mu.Unlock()
59+ return len(r.subs)
60+}
61+
62+func TestAuthorityAdmissionRollback(t *testing.T) {
63+ authority, _, pool, supervisor, _, _ := newTestAuthority(t, 1, nil, immediateConnector)
64+
65+ first, err := authority.AdmitClient(context.Background())
66+ require.NoError(t, err)
67+ var epochFromID ports.BrokerEpoch
68+ for i := 0; i < 8; i++ {
69+ epochFromID = epochFromID<<8 | ports.BrokerEpoch(first.ConnectionID()[i])
70+ }
71+ require.Equal(t, ports.BrokerEpoch(1), epochFromID, "the pool identity must carry the broker epoch")
72+
73+ // Exhaust the pool's connection bound: the second admission must release the
74+ // supervisor client lease it already took.
75+ pool.limits.Clients = 1
76+ _, err = authority.AdmitClient(context.Background())
77+ require.ErrorIs(t, err, ports.BrokerAdmissionLimit)
78+ clients, _, _, _ := supervisorState(supervisor)
79+ require.Equal(t, 1, clients, "a refused admission must release its client lease")
80+ require.Equal(t, 1, poolClientCount(pool), "a refused admission must not leave a pool client")
81+
82+ // A cancelled admission context takes no lease and registers no client.
83+ ctx, cancel := context.WithCancel(context.Background())
84+ cancel()
85+ _, err = authority.AdmitClient(ctx)
86+ require.ErrorIs(t, err, context.Canceled)
87+ clients, _, _, _ = supervisorState(supervisor)
88+ require.Equal(t, 1, clients)
89+
90+ require.NoError(t, first.Close())
91+ clients, _, _, _ = supervisorState(supervisor)
92+ require.Zero(t, clients)
93+ require.Zero(t, poolClientCount(pool))
94+
95+ // A committed supervisor refuses every later admission without taking work.
96+ require.NoError(t, supervisor.Close())
97+ _, err = authority.AdmitClient(context.Background())
98+ requireBrokerClosed(t, err)
99+ clients, _, _, _ = supervisorState(supervisor)
100+ require.Zero(t, clients)
101+}
102+
103+func TestAuthorityAdmissionContextNotRetained(t *testing.T) {
104+ authority, registry, _, _, _, _ := newTestAuthority(t, 1, nil, immediateConnector)
105+ ctx, cancel := context.WithCancel(context.Background())
106+ service, err := authority.AdmitClient(ctx)
107+ require.NoError(t, err)
108+ t.Cleanup(func() { require.NoError(t, service.Close()) })
109+
110+ // The admission context is discarded: the connection keeps serving after the
111+ // caller's setup deadline expires.
112+ cancel()
113+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{registration(t, "user@host:22", 1)}))
114+ require.Len(t, service.Snapshot().Hosts, 1)
115+
116+ stream, err := service.OpenStream(context.Background(), poolRequest(service.ConnectionID(), 1))
117+ require.NoError(t, err)
118+ require.NoError(t, stream.Close())
119+}
120+
121+func TestAuthorityValidation(t *testing.T) {
122+ _, _, pool, supervisor, clock, _ := newTestAuthority(t, 1, nil, immediateConnector)
123+
124+ observing, err := NewRegistry(1, newTestStore(), newTestProbe(1), clock, nil)
125+ require.NoError(t, err)
126+ _, err = NewAuthority(1, observing, pool, supervisor)
127+ require.Error(t, err, "an observing registry must be refused by an offline authority")
128+
129+ disabled, err := NewRegistryWithConfig(1, newTestStore(), nil, clock, nil, RegistryConfig{ObservationDisabled: true})
130+ require.NoError(t, err)
131+ _, err = NewAuthority(2, disabled, pool, supervisor)
132+ require.Error(t, err, "an epoch mismatch must be refused")
133+
134+ _, err = NewAuthority(1, nil, pool, supervisor)
135+ require.Error(t, err, "a missing registry must be refused")
136+ _, err = NewAuthority(1, disabled, nil, supervisor)
137+ require.Error(t, err, "a missing pool must be refused")
138+ _, err = NewAuthority(1, disabled, pool, nil)
139+ require.Error(t, err, "a missing supervisor must be refused")
140+
141+ _, err = NewAuthority(1, disabled, pool, supervisor)
142+ require.NoError(t, err)
143+}
144+
145+func TestServiceOpenStreamScopeFencing(t *testing.T) {
146+ foreign := ports.BrokerConnectionID{0xff}
147+ tests := []struct {
148+ name string
149+ build func(id ports.BrokerConnectionID) ports.BrokerOpenStreamRequest
150+ wantStale bool
151+ wantEpoch bool
152+ }{
153+ {
154+ name: "a foreign connection is fenced as stale",
155+ build: func(ports.BrokerConnectionID) ports.BrokerOpenStreamRequest {
156+ return poolRequest(foreign, 2)
157+ },
158+ wantStale: true,
159+ },
160+ {
161+ name: "a foreign epoch is fenced as a stale epoch",
162+ build: func(id ports.BrokerConnectionID) ports.BrokerOpenStreamRequest {
163+ request := poolRequest(id, 3)
164+ request.Epoch = 2
165+ return request
166+ },
167+ wantEpoch: true,
168+ },
169+ {
170+ name: "an absent identity is filled from the connection",
171+ build: func(ports.BrokerConnectionID) ports.BrokerOpenStreamRequest {
172+ request := poolRequest(ports.BrokerConnectionID{}, 4)
173+ request.Epoch = 0
174+ return request
175+ },
176+ },
177+ }
178+ for _, tc := range tests {
179+ t.Run(tc.name, func(t *testing.T) {
180+ authority, _, _, _, _, _ := newTestAuthority(t, 1, nil, immediateConnector)
181+ service, err := authority.AdmitClient(context.Background())
182+ require.NoError(t, err)
183+ t.Cleanup(func() { require.NoError(t, service.Close()) })
184+
185+ stream, err := service.OpenStream(context.Background(), tc.build(service.ConnectionID()))
186+ switch {
187+ case tc.wantStale:
188+ require.ErrorIs(t, err, ports.BrokerAdmissionStale)
189+ case tc.wantEpoch:
190+ var typed ports.BrokerError
191+ require.ErrorAs(t, err, &typed)
192+ require.Equal(t, ports.BrokerErrorStaleEpoch, typed.Code)
193+ default:
194+ require.NoError(t, err)
195+ require.NoError(t, stream.Close())
196+ }
197+ })
198+ }
199+}
200+
201+func TestServiceCloseStreamScopeFencing(t *testing.T) {
202+ authority, _, _, _, _, _ := newTestAuthority(t, 1, nil, immediateConnector)
203+ service, err := authority.AdmitClient(context.Background())
204+ require.NoError(t, err)
205+ t.Cleanup(func() { require.NoError(t, service.Close()) })
206+ id := service.ConnectionID()
207+
208+ stream, err := service.OpenStream(context.Background(), poolRequest(id, 1))
209+ require.NoError(t, err)
210+
211+ // A wrong-scope close must not retire this connection's stream.
212+ require.ErrorIs(t, service.CloseStream(ports.BrokerConnectionID{0xff}, 1), ports.BrokerAdmissionStale)
213+ select {
214+ case <-stream.Done():
215+ t.Fatal("a wrong-scope close must not retire the stream")
216+ default:
217+ }
218+ require.NoError(t, service.CloseStream(id, 1))
219+ await(t, stream.Done())
220+}
221+
222+func TestServiceSubscriptionCleanup(t *testing.T) {
223+ authority, registry, _, _, _, _ := newTestAuthority(t, 1, nil, immediateConnector)
224+ service, err := authority.AdmitClient(context.Background())
225+ require.NoError(t, err)
226+ t.Cleanup(func() { require.NoError(t, service.Close()) })
227+
228+ sub, err := service.Subscribe()
229+ require.NoError(t, err)
230+ require.Equal(t, 1, registrySubscriptionCount(registry))
231+ sub.Close()
232+ require.Zero(t, registrySubscriptionCount(registry), "an explicit close must release the subscription")
233+
234+ sub, err = service.Subscribe()
235+ require.NoError(t, err)
236+ require.Equal(t, 1, registrySubscriptionCount(registry))
237+
238+ require.NoError(t, service.Close())
239+ require.Zero(t, registrySubscriptionCount(registry), "Close must release tracked subscriptions")
240+ _, err = service.Subscribe()
241+ requireBrokerClosed(t, err)
242+}
243+
244+func TestServiceSnapshotDelegates(t *testing.T) {
245+ authority, registry, _, _, _, _ := newTestAuthority(t, 1, nil, immediateConnector)
246+ service, err := authority.AdmitClient(context.Background())
247+ require.NoError(t, err)
248+ t.Cleanup(func() { require.NoError(t, service.Close()) })
249+
250+ sub, err := service.Subscribe()
251+ require.NoError(t, err)
252+ defer sub.Close()
253+
254+ require.NoError(t, registry.setHosts([]domain.RemoteRegistration{registration(t, "user@host:22", 1)}))
255+ select {
256+ case <-sub.Changed():
257+ case <-time.After(time.Second):
258+ t.Fatal("the connection subscription must observe a registry publication")
259+ }
260+ require.Equal(t, registry.Snapshot(), service.Snapshot())
261+ require.Len(t, service.Snapshot().Hosts, 1)
262+}
263+
264+func TestServiceOperationLeaseLifetime(t *testing.T) {
265+ entered := make(chan struct{})
266+ proceed := make(chan struct{})
267+ authority, _, _, supervisor, _, _ := newTestAuthority(t, 1, nil, func(ctx context.Context, e ports.BrokerResolvedEndpoint) (ports.BrokerPhysicalConnection, error) {
268+ close(entered)
269+ select {
270+ case <-ctx.Done():
271+ return nil, ctx.Err()
272+ case <-proceed:
273+ return &fakePhysical{endpoint: e, done: make(chan struct{})}, nil
274+ }
275+ })
276+ service, err := authority.AdmitClient(context.Background())
277+ require.NoError(t, err)
278+ t.Cleanup(func() { require.NoError(t, service.Close()) })
279+
280+ type opened struct {
281+ stream ports.BrokerLogicalConnection
282+ err error
283+ }
284+ result := make(chan opened, 1)
285+ go func() {
286+ stream, err := service.OpenStream(context.Background(), poolRequest(service.ConnectionID(), 1))
287+ result <- opened{stream: stream, err: err}
288+ }()
289+ <-entered
290+ _, operations, _, _ := supervisorState(supervisor)
291+ require.Equal(t, 1, operations, "open setup must hold one operation lease")
292+
293+ close(proceed)
294+ got := <-result
295+ require.NoError(t, got.err)
296+ require.NotNil(t, got.stream)
297+ _, operations, _, _ = supervisorState(supervisor)
298+ require.Zero(t, operations, "the operation lease must be released once setup completes")
299+ select {
300+ case <-got.stream.Done():
301+ t.Fatal("a stream that opened successfully must outlive its setup lease")
302+ default:
303+ }
304+ require.NoError(t, got.stream.Close())
305+ await(t, got.stream.Done())
306+}
307+
308+func TestServiceCloseOpenRace(t *testing.T) {
309+ for round := 0; round < 25; round++ {
310+ t.Run(fmt.Sprintf("round-%d", round), func(t *testing.T) {
311+ entered := make(chan struct{})
312+ proceed := make(chan struct{})
313+ authority, _, pool, supervisor, _, _ := newTestAuthority(t, 1, nil, func(ctx context.Context, e ports.BrokerResolvedEndpoint) (ports.BrokerPhysicalConnection, error) {
314+ close(entered)
315+ select {
316+ case <-ctx.Done():
317+ return nil, ctx.Err()
318+ case <-proceed:
319+ return &fakePhysical{endpoint: e, done: make(chan struct{})}, nil
320+ }
321+ })
322+ service, err := authority.AdmitClient(context.Background())
323+ require.NoError(t, err)
324+
325+ result := make(chan error, 1)
326+ go func() {
327+ _, err := service.OpenStream(context.Background(), poolRequest(service.ConnectionID(), 1))
328+ result <- err
329+ }()
330+ <-entered
331+
332+ // Close races the parked setup: whichever wins, both must settle
333+ // without deadlock and leave no lease or client behind.
334+ closed := make(chan error, 1)
335+ go func() { closed <- service.Close() }()
336+ close(proceed)
337+ require.NoError(t, <-closed)
338+ <-result
339+
340+ clients, operations, _, _ := supervisorState(supervisor)
341+ require.Zero(t, clients)
342+ require.Zero(t, operations)
343+ require.Zero(t, poolClientCount(pool))
344+ })
345+ }
346+}
347+
348+func TestServiceCloseIsIdempotentAndConcurrent(t *testing.T) {
349+ authority, registry, pool, supervisor, _, _ := newTestAuthority(t, 1, nil, immediateConnector)
350+ service, err := authority.AdmitClient(context.Background())
351+ require.NoError(t, err)
352+ _, err = service.Subscribe()
353+ require.NoError(t, err)
354+ stream, err := service.OpenStream(context.Background(), poolRequest(service.ConnectionID(), 1))
355+ require.NoError(t, err)
356+
357+ const closers = 8
358+ errs := make(chan error, closers)
359+ var wg sync.WaitGroup
360+ for i := 0; i < closers; i++ {
361+ wg.Add(1)
362+ go func() {
363+ defer wg.Done()
364+ errs <- service.Close()
365+ }()
366+ }
367+ wg.Wait()
368+ for i := 0; i < closers; i++ {
369+ require.NoError(t, <-errs)
370+ }
371+ require.NoError(t, service.Close())
372+
373+ require.Zero(t, registrySubscriptionCount(registry))
374+ require.Zero(t, poolClientCount(pool))
375+ clients, _, _, _ := supervisorState(supervisor)
376+ require.Zero(t, clients)
377+ await(t, stream.Done())
378+
379+ _, err = service.OpenStream(context.Background(), poolRequest(service.ConnectionID(), 2))
380+ requireBrokerClosed(t, err)
381+}
382+
383+func TestServiceShutdownTerminatesOwnedWork(t *testing.T) {
384+ authority, _, _, supervisor, _, _ := newTestAuthority(t, 1, nil, immediateConnector)
385+ service, err := authority.AdmitClient(context.Background())
386+ require.NoError(t, err)
387+
388+ stream, err := service.OpenStream(context.Background(), poolRequest(service.ConnectionID(), 1))
389+ require.NoError(t, err)
390+
391+ // Broker shutdown cancels the connection root: owned streams end even before
392+ // the session gets to call Close.
393+ require.NoError(t, supervisor.Close())
394+ await(t, stream.Done())
395+
396+ require.NoError(t, service.Close())
397+ clients, _, _, _ := supervisorState(supervisor)
398+ require.Zero(t, clients)
399+}
400+
401+func TestServiceOfflineMembershipRefusal(t *testing.T) {
402+ authority, registry, _, _, _, _ := newTestAuthority(t, 1, nil, immediateConnector)
403+ service, err := authority.AdmitClient(context.Background())
404+ require.NoError(t, err)
405+ t.Cleanup(func() { require.NoError(t, service.Close()) })
406+
407+ err = service.AddHost(context.Background(), "new@host:22")
408+ require.ErrorIs(t, err, ErrOfflineMembership)
409+ var typed ports.BrokerError
410+ require.ErrorAs(t, err, &typed)
411+
412+ removed, err := service.RemoveHost(context.Background(), "new@host:22")
413+ require.False(t, removed)
414+ require.ErrorIs(t, err, ErrOfflineMembership)
415+
416+ require.Empty(t, registry.Snapshot().Hosts, "a refused mutation must not touch authority")
417+ before := registry.Snapshot()
418+ service.RequestReconcile("new@host:22")
419+ require.Equal(t, before, registry.Snapshot(), "reconcile must be a no-op")
420+}
421+
422+func TestRegistryObservationDisabledRestoresAndPublishes(t *testing.T) {
423+ clock := newManualClock(time.Unix(0, 0))
424+ store := newTestStore()
425+ endpoint := "user@host:22"
426+ store.loaded = ports.BrokerSnapshot{
427+ Epoch: 9,
428+ Revision: 4,
429+ Hosts: []ports.RemoteHostSnapshot{{
430+ Endpoint: endpoint,
431+ Registration: registration(t, endpoint, 1),
432+ Availability: domain.RemoteAvailabilityReachable,
433+ }},
434+ }
435+ r, err := NewRegistryWithConfig(1, store, nil, clock, nil, RegistryConfig{ObservationDisabled: true})
436+ require.NoError(t, err)
437+
438+ // The disabled registry restores durable state under its fresh epoch and
439+ // publishes it without observing anything.
440+ snapshot := r.Snapshot()
441+ require.Equal(t, ports.BrokerEpoch(1), snapshot.Epoch)
442+ require.EqualValues(t, 1, snapshot.Revision)
443+ host, ok := snapshot.Find(endpoint)
444+ require.True(t, ok)
445+ require.Equal(t, domain.RemoteAvailabilityReachable, host.Availability)
446+ require.Zero(t, clock.activeTimers())
447+}
448+
449+func TestRegistryObservationDisabledIssuesNoProbesOrTimers(t *testing.T) {
450+ clock := newManualClock(time.Unix(0, 0))
451+ store := newTestStore()
452+ probe := newTestProbe(4)
453+ r, err := NewRegistryWithConfig(1, store, probe, clock, nil, RegistryConfig{ObservationDisabled: true})
454+ require.NoError(t, err)
455+ require.NoError(t, r.setHosts([]domain.RemoteRegistration{registration(t, "user@host:22", 1)}))
456+
457+ ctx, cancel := context.WithCancel(context.Background())
458+ done := make(chan struct{})
459+ go func() { defer close(done); r.Run(ctx) }()
460+
461+ // A reconcile/probe demand must schedule nothing: no probe starts and no
462+ // timer is armed, so advancing the clock cannot produce observation work.
463+ r.RequestProbe("user@host:22")
464+ clock.Advance(time.Hour)
465+ require.Zero(t, clock.activeTimers())
466+ require.Empty(t, probe.calls)
467+
468+ cancel()
469+ await(t, done)
470+ require.Zero(t, clock.activeTimers())
471+ require.Empty(t, probe.calls)
472+ require.Equal(t, runStopped, r.running.Load())
473+}
474+
475+func TestRegistryObservationDisabledDrainsWriter(t *testing.T) {
476+ clock := newManualClock(time.Unix(0, 0))
477+ store := newBlockingStore()
478+ r, err := NewRegistryWithConfig(1, store, nil, clock, nil, RegistryConfig{ObservationDisabled: true})
479+ require.NoError(t, err)
480+
481+ ctx, cancel := context.WithCancel(context.Background())
482+ runDone := make(chan struct{})
483+ go func() { defer close(runDone); r.Run(ctx) }()
484+
485+ require.NoError(t, r.ReplaceHosts([]ports.BrokerHostRecord{{
486+ Registration: registration(t, "user@host:22", 1),
487+ Pinned: true,
488+ Policy: poolPolicy(),
489+ }}))
490+ select {
491+ case <-store.entered:
492+ case <-time.After(time.Second):
493+ t.Fatal("the durable writer never attempted a write")
494+ }
495+
496+ // Run owns the writer: it must not settle while a durable write is in
497+ // flight, and it must still stop once that write drains.
498+ cancel()
499+ select {
500+ case <-runDone:
501+ t.Fatal("Run must not return before the in-flight durable write drains")
502+ case <-time.After(probeGrace):
503+ }
504+ store.releaseFirst()
505+ await(t, runDone)
506+ require.NotEmpty(t, store.writtenRevisions())
507+ require.Zero(t, clock.activeTimers())
508+}

Keyboard shortcuts

Single-key shortcuts are on. ctrl k and esc work whatever this says.

ctrl k :
go to a section, branch, tag, file or repository
?
this list
j k
move the row cursor down / up
enter
open the row under the cursor; on Files, a directory opens or closes in place
↑ ↓ ← → home end
in the file tree (focus on a row): previous / next row, close / open a directory or go to its parent / first entry, first / last row. They never open a file
l h
on Files: open a directory or a file / close a directory or go up to its parent, then to the parent page. Elsewhere: into the side panel (on a merge request, its files) / back to the content; there j k move between its controls
g g G
first / last row
/
filter the list; on Files, find a file anywhere in the repository (esc clears, enter opens the best match)
b
switch branch or tag
g s g f g c g b g m g i g r g a
tabs: summary, files, commits, branches & tags, merge requests, issues, releases, actions (when available)
g h
home
w
wide mode: use the whole width of the screen, or not (remembered)
y
copy a permalink pinned to the commit
d u x
diff: split / unified / collapse or expand all
e E
diff: the whole file in view / every file, with the lines the hunks hide
n
merge request: next note in the diff
esc
close a dialog, leave a field

Branches 12

Tags 11

↑↓ moveenter openesc close