detached 3e0d64d

feat(daemon): preserve service across session purges

commit
3e0d64d401f59ed7cfdb7047b441cde2eed9640f
author
brice · · 2026-09-17 07:18 UTC
parent
0a1c4c5
signature
GPG VerifiedCD88F9E564E0102646FB6077A1A07C7C7C3E349D · arch-gpg Verified for the committer email
36 files · +4505 -259
Unified
UnifiedSplit
M AGENTS.md +3 -1
@@ -77,7 +77,9 @@ Typed messages and negotiated version live in `internal/protocol`. Remote discov
7777
7878 - Ephemeral numbered sessions survive detach while the daemon retains them, but are not persisted.
7979 - Named sessions survive headless and persist across daemon restarts.
80-- The daemon starts on first use and exits when the last session ends.
80+- The daemon starts on first use and survives an empty session registry: final session removal is not shutdown, so it can be reused; it ends only on an explicit shutdown request or process cancellation.
81+- A bulk purge (`kill --all`) takes a transient admission gate and bounds each phase with its own deadline: admission drain, stopped/broken sweep, and a shared live-teardown budget. A repository delete that ignores cancellation is detached behind an exact name/incarnation/created-time fence and reported as a typed failure instead of holding purge admission or control.
82+- A live unit whose teardown aborts before destructive ownership stays registered, and its snapshot-coordinator quarantine is rolled back so the surviving session resumes checkpoint scheduling and its attached clients are unfrozen.
8183 - Each connection has a 15-second handshake budget from connect through the initial committed publication.
8284 - Local and remote attach use the same typed `Hello`/`Welcome` session protocol. `sessionwire` translates typed traffic to Protobuf envelopes carried by IPC, QUIC, or SSH stdio.
8385 - Remote attach bootstraps an authenticated direct QUIC connection by default; `VEV_REMOTE_TRANSPORT=stdio` explicitly selects an SSH-only carriage.
M README.md +1 -1
@@ -42,7 +42,7 @@ vev --ui-observe ... expose passive observation for an attach
4242 vev --ui-control ... expose observation and input control
4343 ```
4444
45-The daemon starts on first use and exits after the last session. Numbered sessions survive detach; named sessions also survive daemon restarts. Exiting the final shell removes that session and returns to the most recently used session; with no previous session, vev exits. See [durable-session recovery](docs/durable-session-recovery.md).
45+The daemon starts on first use and survives an empty session registry: removing the last session is not shutdown, so it can be reused. `kill --all` purges every session and leaves the daemon running; it ends only on an explicit `kill --daemon` or process cancellation. Numbered sessions survive detach; named sessions also survive daemon restarts. Exiting the final shell removes that session and returns to the most recently used session; with no previous session, vev exits. See [durable-session recovery](docs/durable-session-recovery.md).
4646
4747 > [!NOTE]
4848 > A release that bumps vev's protocol version resets named sessions saved by an older protocol. Session names are retained, but layouts, tabs, terminal history, recovery transcripts, and process-recovery state are discarded. Each local or remote daemon applies this reset to its own sessions when it first starts with the new protocol.
M docs/architecture.md +8 -0
@@ -106,6 +106,14 @@ Attention is a daemon fact, not an overlay: a bell raised by a PTY, by a status-
106106
107107 Remote discovery has two owners with separated writers. The daemon composes its own store, catalogue client, cache, and monitor for the structured rows and exact-target validation its picker serves. The launching client composes a runner-scoped `remotes.HostRegistry` over its own store, catalogue client, and a runner-local catalogue cache that seeds from the durable snapshot and never writes it, so one monitor can never truncate the other's file. The registry projects that discovery through `ports.RemoteDirectory` and resolves each endpoint once into a `ports.RemoteEndpointBinding` — the typed dialer every session on that endpoint attaches through, plus the environment to advertise — which the runner reuses for its initial remote attach and for every later handoff. Transport mode, launch allowlist, and endpoint environment stay in `internal/app` behind `ports.RemoteEndpointFactory`; the client never inspects them, and a resolution failure is never cached, so a refused endpoint fails the handoff instead of falling back to another resolver.
108108
109+Current state: the daemon-side monitor is deliberately still active. `Serve`
110+starts the runner through `daemon.WithRemoteMonitor` and joins it inside the
111+bounded remote-shutdown budget, and the daemon keeps its own remote directory
112+and host registry. That daemon-owned monitoring (and the host registry that
113+feeds it) belongs to the coordinated P7 removal set; it is not removed,
114+gated, or polled differently before that cutover, and no polling gate is
115+introduced here (GO-001, deferred to P7).
116+
109117 ## Session composition
110118
111119 ```text
M docs/configuration.md +3 -1
@@ -179,7 +179,9 @@ Remote host commands, listing, and successful direct-attach learning are always
179179
180180 Set `VEV_LOG=debug`, `VEV_LOG=warn`, or `VEV_LOG=error` to change verbosity; the default is `info`. JSON-line logs such as `vev-daemon.log` live in `$XDG_STATE_HOME/vev`, or `~/.local/state/vev` when unset. The same state directory contains the strict session catalogue, any private pre-migration catalogue backup, notices, `hosts.json`, and `snapshots/`. The lifecycle lock and socket live in `$XDG_RUNTIME_DIR/vev` (with platform runtime fallbacks).
181181
182-Recovery events include `lifecycle_owner_wait`, `lifecycle_owner_acquired`, `lifecycle_owner_released`, `catalogue_validated`, `catalogue_compaction_recovery_complete`, `session_restore_complete`, `fallback_checkpoint_promoted`, `snapshot_head_repair_complete`, `session_degraded`, `snapshot_maintenance_progress`, `interrupted_transaction_recovery_complete`, and `daemon_startup_complete`.
182+Recovery events include `lifecycle_owner_wait`, `lifecycle_owner_acquired`, `lifecycle_owner_released`, `catalogue_validated`, `catalogue_compaction_recovery_complete`, `session_restore_complete`, `fallback_checkpoint_promoted`, `snapshot_head_repair_complete`, `session_degraded`, `snapshot_garbage_collection_complete`, `interrupted_transaction_recovery_complete`, and `daemon_startup_complete`.
183+
184+Snapshot garbage collection is startup-only: the daemon runs one coordinator-owned pass before it publishes its socket, and no periodic snapshot collection runs afterward. The five-second idle maintenance loop compresses sealed scrollback pages and never collects snapshots or incarnations (GO-002, deferred).
183185
184186 Catalogue failure is fail-closed: vev does not publish an empty replacement daemon. Preserve the state directory, inspect `catalogue_validation_failed`, correct storage or ownership problems, and retry without editing catalogue files. See [Durable session recovery](durable-session-recovery.md) for explicit recovery commands, migration, diagnostics, and the committed checkpoint plus up to two direct fallbacks retention policy.
185187
M docs/durable-session-recovery.md +4 -0
@@ -6,6 +6,10 @@ vev opens named-session state only while holding `$XDG_RUNTIME_DIR/vev/lifecycle
66
77 `stopped` is safe fresh metadata, `restoring` is validation in progress, and `degraded` preserves uncertain state without starting a replacement shell. A degraded session remains visible in `vev ls`, but attach is refused until an explicit recovery action succeeds. A session reserved for durable purge is hidden from listings and pickers until deletion either completes or leaves a retained broken record.
88
9+## Bulk purge
10+
11+`kill --all` purges every live, stopped, and broken session while leaving the daemon running. Its admission gate is transient and each phase spends its own bounded deadline (admission drain, stopped/broken sweep, shared live teardown), so an uncooperative repository can never hold purge admission or control indefinitely. A durable delete that ignores cancellation is completed by a detached worker under the exact session name, incarnation, and creation time it captured, so it can never remove a same-name replacement; until it returns, the stopped authority still reserves the name and the purge reports a typed failure. A live unit whose teardown aborts before it owns destructive work stays registered, and its snapshot-coordinator quarantine is rolled back so it keeps checkpointing.
12+
913 ## Recovery commands
1014
1115 - `vev cmd -s NAME session-recovery discard`
M docs/protocol.md +9 -0
@@ -188,6 +188,15 @@ while `pickerOpen`; only typed picker messages act.
188188 and carry no async byte-budget charge; async sends are charged on
189189 admission and released exactly once on dequeue.
190190
191+Current state: a one-shot control client uses the daemon closing the
192+connection (EOF) as its success signal, so a control failure whose error
193+response cannot be delivered would otherwise be indistinguishable from
194+success; the daemon only logs such a failed send
195+(`daemon.logControlSendFailure`) before closing. The wire-level decision for that
196+residual belongs to the P7 explicit result protocol, which replaces close-as-
197+success with an explicit per-request result; until that cutover the close-based
198+convention is kept unchanged (GO-005, deferred to P7).
199+
191200 ## Limits
192201
193202 | Item | Ceiling | Owner |
M internal/app/daemon_integration_test.go +256 -26
@@ -150,6 +150,34 @@ func killDaemon(dir string) error {
150150 return requestKill(dir, protocol.KillDaemon)
151151 }
152152
153+// killNamedSession deletes one named live or stopped session over the control
154+// connection, using the same wire scope the client sends for `kill -s NAME`.
155+func killNamedSession(dir, name string) error {
156+ raw, err := ipc.DialContext(context.Background(), dir)
157+ if err != nil {
158+ return err
159+ }
160+ conn := sessionwire.NewClientConnection(raw)
161+ defer func() { _ = conn.Close() }()
162+ if err := conn.SendClient(protocol.Kill{Scope: protocol.KillSession, Name: name}); err != nil {
163+ return err
164+ }
165+ reply, err := receiveKillReply(conn, daemonStopTimeout)
166+ if errors.Is(err, context.DeadlineExceeded) {
167+ return err
168+ }
169+ if errors.Is(err, io.EOF) {
170+ return nil
171+ }
172+ if err != nil {
173+ return err
174+ }
175+ if em, ok := reply.(protocol.ErrorMsg); ok {
176+ return fmt.Errorf("kill failed: %s", em.Text)
177+ }
178+ return nil
179+}
180+
153181 func requestKill(dir string, scope protocol.KillScope) error {
154182 raw, err := ipc.DialContext(context.Background(), dir)
155183 if err != nil {
@@ -333,6 +361,27 @@ func awaitText(t *testing.T, p *typedPump, sz domain.Size, want string) {
333361 _ = awaitScreenText(t, p, sz, want)
334362 }
335363
364+// awaitDetached consumes typed messages until the daemon signals the session's
365+// end with the expected Detached reason, ignoring interleaved Output frames.
366+func awaitDetached(t *testing.T, p *typedPump, reason uint8) {
367+ t.Helper()
368+ timeout := time.After(5 * time.Second)
369+ for {
370+ select {
371+ case message, ok := <-p.ch:
372+ if !ok {
373+ t.Fatal("connection closed before Detached")
374+ }
375+ if detached, ok := message.(protocol.Detached); ok {
376+ require.Equal(t, reason, detached.Reason)
377+ return
378+ }
379+ case <-timeout:
380+ t.Fatal("timed out waiting for Detached")
381+ }
382+ }
383+}
384+
336385 // awaitScreenText is like awaitText, but returns the reconstructed screen text
337386 // at the point the wanted text appears so callers can make additional checks.
338387 func awaitScreenText(t *testing.T, p *typedPump, sz domain.Size, want string) string {
@@ -597,12 +646,13 @@ func TestIntegration_EphemeralSurvivesDetachAndReattaches(t *testing.T) {
597646 defer func() { _ = tr2.Close() }()
598647 awaitText(t, p2, sz, "MARKER")
599648
600- require.NoError(t, killAll(dir))
649+ // An ephemeral session's daemon is ended only by the explicit daemon stop.
650+ require.NoError(t, killDaemon(dir))
601651 select {
602652 case err := <-served:
603653 require.NoError(t, err)
604654 case <-time.After(5 * time.Second):
605- t.Fatal("daemon did not shut down after kill all")
655+ t.Fatal("daemon did not stop after kill --daemon")
606656 }
607657 }
608658
@@ -679,12 +729,20 @@ func TestIntegration_KillDaemonPreservesMultipleNamedSessions(t *testing.T) {
679729 {Name: "beta", State: protocol.SessionDown},
680730 }, sessions.Sessions)
681731
732+ // Kill-all purges every durable record but leaves this daemon serving.
682733 require.NoError(t, killAll(dir))
734+ require.Empty(t, listRemoteSessions(t, dir).Sessions)
735+ select {
736+ case err := <-served:
737+ t.Fatalf("kill-all stopped the restarted daemon: %v", err)
738+ default:
739+ }
740+ require.NoError(t, killDaemon(dir))
683741 select {
684742 case err := <-served:
685743 require.NoError(t, err)
686744 case <-time.After(5 * time.Second):
687- t.Fatal("restarted daemon did not stop after kill all")
745+ t.Fatal("restarted daemon did not stop after kill --daemon")
688746 }
689747
690748 served = start()
@@ -721,6 +779,142 @@ func TestIntegration_NamedSurvivesReattach(t *testing.T) {
721779 awaitText(t, p2, sz, "MARKER")
722780 }
723781
782+// TestIntegration_NamedSessionLastRemovalKeepsDaemonServing is the P4.1
783+// app-level contract (P4.1 review GO-003): a named session killed as the last
784+// registry entry is removed without stopping Serve, and the same daemon process
785+// accepts the same name again. It runs over the real socket, PTY, and
786+// catalogue/recovery/snapshot persistence the production composition installs.
787+// A named session's final live removal is a durable purge, so the re-attached
788+// name carries a fresh persisted incarnation rather than resuming the deleted
789+// one.
790+func TestIntegration_NamedSessionLastRemovalKeepsDaemonServing(t *testing.T) {
791+ sz := domain.Size{Cols: 80, Rows: 24}
792+ stateDir := filepath.Join(t.TempDir(), "state")
793+ dir := filepath.Join(t.TempDir(), "runtime")
794+ repository := snapshot.NewRepository(filepath.Join(stateDir, "snapshots"))
795+ opened, err := persist.OpenOrCreate(stateDir)
796+ require.NoError(t, err)
797+ coordinator := recovery.NewCoordinator(opened.Catalogue, repository, rand.Reader)
798+
799+ _, served := startDaemonInDir(t, dir,
800+ daemon.WithShell("/bin/cat", nil),
801+ daemon.WithCatalogue(opened.Catalogue, opened.Records),
802+ daemon.WithSnapshotRepository(repository),
803+ daemon.WithRecoveryCoordinator(coordinator),
804+ )
805+
806+ first, p1 := attach(t, dir, protocol.IntentNew, "work", sz)
807+ require.NoError(t, first.SendClient(protocol.Input{Data: []byte("ping\n")}))
808+ awaitText(t, p1, sz, "ping")
809+ firstRecord, ok, err := opened.Catalogue.Record("work")
810+ require.NoError(t, err)
811+ require.True(t, ok, "a live named session must own a durable record")
812+
813+ // End it as the last session: the registry empties, but that is not daemon
814+ // shutdown in the P4.1 model.
815+ require.NoError(t, killNamedSession(dir, "work"))
816+ awaitDetached(t, p1, protocol.ReasonSessionKilled)
817+ require.Eventually(t, func() bool { return len(listRemoteSessions(t, dir).Sessions) == 0 },
818+ 5*time.Second, 10*time.Millisecond, "session was not removed")
819+ select {
820+ case err := <-served:
821+ t.Fatalf("Serve stopped after the last session was removed: %v", err)
822+ default:
823+ }
824+ _, ok, err = opened.Catalogue.Record("work")
825+ require.NoError(t, err)
826+ require.False(t, ok, "final live removal purges the durable record")
827+
828+ // The same daemon process accepts the same name again and can still serve a
829+ // fresh terminal operation.
830+ second, p2 := attach(t, dir, protocol.IntentNew, "work", sz)
831+ require.NoError(t, second.SendClient(protocol.Input{Data: []byte("pong\n")}))
832+ awaitText(t, p2, sz, "pong")
833+ sessions := listRemoteSessions(t, dir).Sessions
834+ require.Len(t, sessions, 1)
835+ require.Equal(t, "work", sessions[0].Name)
836+ require.Equal(t, protocol.SessionUp, sessions[0].State)
837+ secondRecord, ok, err := opened.Catalogue.Record("work")
838+ require.NoError(t, err)
839+ require.True(t, ok, "re-attached named session must persist its identity")
840+ require.NotEqual(t, firstRecord.IncarnationID, secondRecord.IncarnationID,
841+ "a re-created name must not reuse the purged incarnation")
842+
843+ require.NoError(t, second.Close())
844+ require.NoError(t, killDaemon(dir))
845+ select {
846+ case err := <-served:
847+ require.NoError(t, err)
848+ case <-time.After(5 * time.Second):
849+ t.Fatal("daemon did not stop after kill --daemon")
850+ }
851+}
852+
853+// TestIntegration_NamedSessionRestoresPersistedIdentity proves a stopped named
854+// session is restored inside the daemon process that is already serving it,
855+// keeping its exact persisted identity. The seed daemon stops while holding the
856+// session as its only registry entry, which preserves it as stopped authority;
857+// the daemon under test then resumes it over a real socket with the production
858+// catalogue/recovery/snapshot options and serves a fresh operation.
859+func TestIntegration_NamedSessionRestoresPersistedIdentity(t *testing.T) {
860+ sz := domain.Size{Cols: 80, Rows: 24}
861+ stateDir := filepath.Join(t.TempDir(), "state")
862+ repository := snapshot.NewRepository(filepath.Join(stateDir, "snapshots"))
863+ name := publishRestorableCheckpoint(t, stateDir, repository)
864+
865+ opened, err := persist.OpenOrCreate(stateDir)
866+ require.NoError(t, err)
867+ stopped, ok, err := opened.Catalogue.Record(name)
868+ require.NoError(t, err)
869+ require.True(t, ok, "stopped named session must keep its durable record")
870+ require.NotNil(t, stopped.Committed, "stopped named session must keep its committed checkpoint")
871+
872+ dir := filepath.Join(t.TempDir(), "runtime")
873+ coordinator := recovery.NewCoordinator(opened.Catalogue, repository, rand.Reader)
874+ _, served := startDaemonInDir(t, dir,
875+ daemon.WithShell("/bin/cat", nil),
876+ daemon.WithCatalogue(opened.Catalogue, opened.Records),
877+ daemon.WithSnapshotRepository(repository),
878+ daemon.WithRecoveryCoordinator(coordinator),
879+ )
880+ require.Equal(t, []protocol.SessionInfo{{Name: name, State: protocol.SessionDown}}, listRemoteSessions(t, dir).Sessions)
881+ select {
882+ case err := <-served:
883+ t.Fatalf("Serve stopped before the restore: %v", err)
884+ default:
885+ }
886+
887+ conn, p := attach(t, dir, protocol.IntentAttach, name, sz)
888+ require.NoError(t, conn.SendClient(protocol.Input{Data: []byte("revived\n")}))
889+ awaitText(t, p, sz, "revived")
890+ require.Eventually(t, func() bool {
891+ sessions := listRemoteSessions(t, dir).Sessions
892+ return len(sessions) == 1 && sessions[0].State == protocol.SessionUp
893+ }, 5*time.Second, 10*time.Millisecond, "named session was not restored")
894+
895+ restored, ok, err := opened.Catalogue.Record(name)
896+ require.NoError(t, err)
897+ require.True(t, ok)
898+ require.Equal(t, stopped.IncarnationID, restored.IncarnationID,
899+ "restore must retain the persisted incarnation")
900+ require.Equal(t, stopped.CreatedAt, restored.CreatedAt,
901+ "restore must retain the persisted lifecycle timestamp")
902+
903+ require.NoError(t, conn.Close())
904+ select {
905+ case err := <-served:
906+ t.Fatalf("Serve stopped while the restored session was live: %v", err)
907+ default:
908+ }
909+ require.NoError(t, killDaemon(dir))
910+ select {
911+ case err := <-served:
912+ require.NoError(t, err)
913+ case <-time.After(5 * time.Second):
914+ t.Fatal("daemon did not stop after kill --daemon")
915+ }
916+}
917+
724918 func TestMultipleClientsOneLifecycleOwner(t *testing.T) {
725919 dir := filepath.Join(t.TempDir(), "runtime")
726920 owner, err := lifecycle.TryAcquire(dir)
@@ -1495,31 +1689,67 @@ func (*integrationTransport) Close() error { return nil }
14951689 func (*integrationTransport) LocalAddr() net.Addr { return nil }
14961690 func (*integrationTransport) RemoteAddr() net.Addr { return nil }
14971691
1498-func TestIntegration_KillAllShutsDownDaemon(t *testing.T) {
1692+// TestIntegration_KillAllPurgesSessionsAndKeepsDaemon is the P4.2 app-level
1693+// contract: `kill --all` removes every live and stopped session but leaves the
1694+// daemon serving, and the same process accepts fresh named and ephemeral
1695+// sessions that start PTYs, forward input, and open a tab through the ordinary
1696+// command palette. Only the distinct explicit daemon stop ends it.
1697+func TestIntegration_KillAllPurgesSessionsAndKeepsDaemon(t *testing.T) {
14991698 sz := domain.Size{Cols: 80, Rows: 24}
1500- dir, served := startDaemon(t, daemon.WithShell("/bin/sh", []string{"-c", "sleep 30"}))
1501-
1502- tr1, _ := attach(t, dir, protocol.IntentNew, "one", sz)
1503- require.NoError(t, tr1.Close())
1504- tr2, _ := attach(t, dir, protocol.IntentNew, "two", sz)
1505- require.NoError(t, tr2.Close())
1506-
1507- killRaw, err := ipc.DialContext(context.Background(), dir)
1508- require.NoError(t, err)
1509- killTr := sessionwire.NewClientConnection(killRaw)
1510- defer func() { _ = killTr.Close() }()
1511- require.NoError(t, killTr.SendClient(protocol.Kill{Scope: protocol.KillAll}))
1699+ dir, served := startDaemon(t, daemon.WithShell("/bin/cat", nil))
1700+
1701+ first, firstPump := attach(t, dir, protocol.IntentNew, "one", sz)
1702+ require.NoError(t, first.SendClient(protocol.Input{Data: []byte("alpha\n")}))
1703+ awaitText(t, firstPump, sz, "alpha")
1704+ // A detached named session becomes stopped authority, so kill-all must purge
1705+ // both the live and the stopped record in one operation.
1706+ second, _ := attach(t, dir, protocol.IntentNew, "two", sz)
1707+ require.NoError(t, second.Close())
1708+ require.Len(t, listRemoteSessions(t, dir).Sessions, 2)
15121709
1513- _, err = killTr.ReceiveServer()
1514- require.ErrorIs(t, err, io.EOF)
1515- var serveErr error
1710+ require.NoError(t, killAll(dir))
1711+ // The daemon survives the purge: Serve has not returned and the registry is
1712+ // empty.
1713+ select {
1714+ case err := <-served:
1715+ t.Fatalf("KillAll stopped Serve: %v", err)
1716+ default:
1717+ }
1718+ require.Eventually(t, func() bool { return len(listRemoteSessions(t, dir).Sessions) == 0 },
1719+ 5*time.Second, 10*time.Millisecond, "kill-all did not purge every session")
1720+
1721+ // Reuse: a fresh named session starts its PTY and forwards input.
1722+ third, thirdPump := attach(t, dir, protocol.IntentNew, "again", sz)
1723+ require.NoError(t, third.SendClient(protocol.Input{Data: []byte("beta\n")}))
1724+ awaitText(t, thirdPump, sz, "beta")
1725+ // A fresh ephemeral session is admitted too.
1726+ fourth, fourthPump := attach(t, dir, protocol.IntentEphemeral, "", sz)
1727+ require.NoError(t, fourth.SendClient(protocol.Input{Data: []byte("gamma\n")}))
1728+ awaitText(t, fourthPump, sz, "gamma")
1729+
1730+ // A tab opened after the purge is real, live topology.
1731+ require.NoError(t, third.SendClient(protocol.Input{Data: []byte("\x1b ")}))
1732+ awaitText(t, thirdPump, sz, "Commands")
1733+ require.NoError(t, third.SendClient(protocol.Input{Data: []byte("CNT\r")}))
15161734 require.Eventually(t, func() bool {
1517- select {
1518- case serveErr = <-served:
1519- return true
1520- default:
1521- return false
1735+ for _, info := range listRemoteSessions(t, dir).Sessions {
1736+ if info.Name == "again" {
1737+ return info.Tabs == 2
1738+ }
15221739 }
1523- }, 15*time.Second, 10*time.Millisecond, "daemon did not shut down after kill all")
1524- require.NoError(t, serveErr)
1740+ return false
1741+ }, 5*time.Second, 10*time.Millisecond, "a tab created after the purge was not published")
1742+
1743+ _ = first.Close()
1744+ require.NoError(t, third.Close())
1745+ require.NoError(t, fourth.Close())
1746+
1747+ // Only the distinct explicit daemon stop ends the daemon.
1748+ require.NoError(t, killDaemon(dir))
1749+ select {
1750+ case err := <-served:
1751+ require.NoError(t, err)
1752+ case <-time.After(5 * time.Second):
1753+ t.Fatal("daemon did not stop after kill --daemon")
1754+ }
15251755 }
M internal/app/run.go +6 -8
@@ -135,7 +135,7 @@ usage:
135135 vev host rm <host> remove a pinned remote host
136136 vev host list list known remote hosts
137137 vev kill <name> kill a session
138- vev kill --all kill all sessions and stop the daemon
138+ vev kill --all kill all sessions (the daemon keeps running)
139139 vev kill --daemon stop the active vev daemon
140140 vev cmd <command> run a control command (vev cmd --help)
141141 vev --ui-observe expose passive observation for this interactive client
@@ -1771,16 +1771,14 @@ func runKill(ctx context.Context, name string, all, daemon bool) (retErr error)
17711771 return cause
17721772 }
17731773 }
1774- if all || daemon {
1774+ // Only an explicit daemon stop ends the daemon; kill-all purges sessions and
1775+ // leaves it running, so it must not wait for an ownership transfer.
1776+ if daemon {
17751777 waitCtx, cancel := context.WithTimeout(ctx, daemonStopTimeout)
17761778 defer cancel()
17771779 owner, waitErr := waitForLifecycleAvailability(waitCtx, ipc.SocketDir(), defaultBackoff)
17781780 if waitErr != nil {
1779- cause := fmt.Errorf("vev: waiting for daemon ownership transfer: %w", waitErr)
1780- if daemon {
1781- return forceStopDaemonFallback(ctx, cause)
1782- }
1783- return cause
1781+ return forceStopDaemonFallback(ctx, fmt.Errorf("vev: waiting for daemon ownership transfer: %w", waitErr))
17841782 }
17851783 if releaseErr := owner.Release(); releaseErr != nil {
17861784 return fmt.Errorf("vev: releasing daemon ownership probe: %w", releaseErr)
@@ -1796,7 +1794,7 @@ func printKillSuccess(name string, all, daemon bool) {
17961794 return
17971795 }
17981796 if all {
1799- fmt.Println("killed all sessions and stopped daemon")
1797+ fmt.Println("killed all sessions")
18001798 return
18011799 }
18021800 fmt.Printf("killed %s\n", name)
M internal/usecase/daemon/attachment_effect_gate_test.go +4 -1
@@ -381,7 +381,10 @@ func TestKillSessionInterruptsCapturedSendWithoutClosingNewerIncarnation(t *test
381381 ac.replaceTransport(fresh)
382382 }
383383
384- require.NoError(t, d.killSession(sess, protocol.ReasonSessionKilled, false))
384+ // The transport incarnation changed after the snapshot, so the kill aborts
385+ // before publishing terminal ownership. That abort must be observable: a
386+ // silent nil would report a teardown that never happened as success.
387+ require.ErrorIs(t, d.killSession(sess, protocol.ReasonSessionKilled, false), errSessionKillParticipantsChanged)
385388 awaitTestCompletion(t, sendDone, "captured incarnation attachment effect ticket did not retire")
386389 require.True(t, stale.Closed(), "exact captured in-flight transport was not interrupted")
387390 require.False(t, fresh.Closed(), "newer transport incarnation was interrupted")
M internal/usecase/daemon/client.go +9 -8
@@ -9,6 +9,15 @@
99 // sessionInteractivelyAttachedLocked), but must never freeze, drain, or wait on
1010 // a gate while holding any architecture lock. Freeze/drain always runs with
1111 // every architecture lock released.
12+//
13+// Daemon-level gate order is moveLifecycleMu before d.mu: reserveMoveLifecycles
14+// holds moveLifecycleMu while taking d.mu to reserve the KillAll purge-gate
15+// counter, and releaseMoveAdmissionLocked does the same while releasing it.
16+// closeMoveLifecycles deliberately drops moveLifecycleMu before taking d.mu, so
17+// no path holds d.mu and then waits on moveLifecycleMu. Consequently
18+// moveLifecycleReservation.Release must never be called with d.mu held: it takes
19+// moveLifecycleMu first, releases both the move-gate and purge-gate reservations
20+// in that order, and only then drops moveLifecycleMu.
1221 package daemon
1322
1423 import (
@@ -1120,11 +1129,3 @@ func (d *Daemon) recordClientNotice(sess *session, ac *attachedClient, sev domai
11201129 func (d *Daemon) resizeForFirstPaint(sess *session, ac *attachedClient, sz domain.Size) bool {
11211130 return sess.geometry.requestResize(d, sess, ac, sz, true)
11221131 }
1123-
1124-// killSession removes a session and tears down its resources. It is
1125-// idempotent: only the caller that wins the registry delete acts. When the
1126-// registry empties it marks the daemon closing (atomically with the
1127-// empty-check, under d.mu) and signals shutdown.
1128-//
1129-// Teardown ordering matters: context cancel, pty.Close, and the done signal
1130-// run first and unconditionally — never gated behind a client send. The
M internal/usecase/daemon/coderabbit_regression_test.go +3 -3
@@ -29,7 +29,7 @@ func TestRestoreAmbiguousBadVersionLoadFailurePreservesCheckpoint(t *testing.T)
2929 }
3030 d, catalogue := newDurableRecoveryDaemon(t, []domain.CatalogueRecord{record}, repository)
3131
32- d.restoreCatalogue(context.Background(), mustDurableRecords(t, catalogue))
32+ d.restoreIncrementalSnapshots(context.Background())
3333
3434 persisted, ok, err := catalogue.Record(record.Name)
3535 require.NoError(t, err)
@@ -54,7 +54,7 @@ func TestRestoreTransientLoadFailureBecomesDiscardable(t *testing.T) {
5454 }
5555 d, catalogue := newDurableRecoveryDaemon(t, []domain.CatalogueRecord{record}, repository)
5656
57- d.restoreCatalogue(context.Background(), mustDurableRecords(t, catalogue))
57+ d.restoreIncrementalSnapshots(context.Background())
5858
5959 persisted, ok, err := catalogue.Record(record.Name)
6060 require.NoError(t, err)
@@ -177,7 +177,7 @@ func TestStaleIncompatibleRestoreCannotOverwriteNewerAuthority(t *testing.T) {
177177 d.setStoppedRecovery(replacement, tt.state)
178178 }
179179
180- d.restoreCatalogue(context.Background(), mustDurableRecords(t, catalogue))
180+ d.restoreIncrementalSnapshots(context.Background())
181181
182182 persisted, ok, err := catalogue.Record(record.Name)
183183 require.NoError(t, err)
M internal/usecase/daemon/control.go +22 -0
@@ -87,6 +87,15 @@ func (d *Daemon) boundedControlSend(tr ports.ServerConnection, message protocol.
8787 return err
8888 }
8989
90+// logControlSendFailure records that an error response could not be delivered.
91+// A one-shot control client treats EOF as success, so a failed error send would
92+// otherwise let it infer success from a silent close.
93+func (d *Daemon) logControlSendFailure(operation string, err error) {
94+ if err != nil {
95+ d.log.Warn("control error response send failed", "operation", operation, "err", err)
96+ }
97+}
98+
9099 func serverCommandResult(result protocol.CommandResult) protocol.CommandResult { return result }
91100
92101 func (d *Daemon) dispatchCommand(ctx context.Context, request protocol.CommandRequest) protocol.CommandResult {
@@ -148,6 +157,16 @@ func (d *Daemon) runControl(cmd command.Command, exec controlExec, request proto
148157 return commandFailure(protocol.ErrInvalidCommandArgs, "usage: "+cmd.Usage)
149158 case errors.Is(err, errSessionNameInUse):
150159 return commandFailure(protocol.ErrNameTaken, err.Error())
160+ case errors.Is(err, errPurgeAdmissionClosed):
161+ // A KillAll purge owns the exact lifecycle set. The operation is
162+ // transient (the daemon is not shutting down), so surface a retryable
163+ // internal failure rather than a name or shutdown verdict.
164+ return commandFailure(protocol.ErrInternal, err.Error())
165+ case errors.Is(err, errSessionKillParticipantsChanged):
166+ // A concurrent detach or token resume invalidated the teardown snapshot
167+ // before it removed anything. The operation is safe to retry, so report
168+ // that clear outcome instead of the internal sentinel text.
169+ return commandFailure(protocol.ErrInternal, sessionKillRetryMessage)
151170 case isMoveCommandError(err):
152171 return moveCommandFailure(err)
153172 case errors.Is(err, layout.ErrNotInSplit):
@@ -490,6 +509,9 @@ func (e controlExec) CreateSessionNamed(name string) error {
490509 if e.d.closing {
491510 return errors.New("daemon is shutting down")
492511 }
512+ if e.d.purgeAdmissionClosedLocked() {
513+ return errPurgeAdmissionClosed
514+ }
493515 if e.d.nameLiveOrStoppedLocked(name) {
494516 return errSessionNameInUse
495517 }
M internal/usecase/daemon/control_test.go +85 -0
@@ -995,6 +995,25 @@ func TestHandleCommandMovePaneUsesActiveFocusedSourceWithSessionFlag(t *testing.
995995 destTab.mu.Unlock()
996996 }
997997
998+// TestHandleCommandMovePanePurgeAdmissionMapsToRetryableError proves a move
999+// command rejected because a transient KillAll purge owns the exact lifecycle
1000+// set surfaces the same retryable internal outcome as the create path, instead
1001+// of misreporting the destination as gone.
1002+func TestHandleCommandMovePanePurgeAdmissionMapsToRetryableError(t *testing.T) {
1003+ d := newTestDaemon(t, nil, stubClock{})
1004+ addControlSession(d, "work", "t_work", "p_work")
1005+ addNamedMoveDestination(d, "dest", "t_dest", "p_dest")
1006+
1007+ beginTestPurgeAdmission(t, d)
1008+ result := sendCommand(t, d, protocol.CommandRequest{
1009+ Slug: "move-pane", Args: []string{"dest", "t_dest"}, TargetSession: "work",
1010+ })
1011+ require.False(t, result.OK)
1012+ require.Equal(t, protocol.ErrInternal, result.Code)
1013+ require.Equal(t, errPurgeAdmissionClosed.Error(), result.Text)
1014+ d.endPurgeAdmission()
1015+}
1016+
9981017 func TestHandleCommandMovePaneSelfUsesStableSourceIDs(t *testing.T) {
9991018 d := newTestDaemon(t, nil, stubClock{})
10001019 source := addControlSession(d, "work", "t_active", "p_active")
@@ -1441,3 +1460,69 @@ func addControlSession(d *Daemon, name, tabID, paneID string) *session {
14411460 d.mu.Unlock()
14421461 return sess
14431462 }
1463+
1464+// TestHandleKillSessionParticipantsChangedReportsRetryable proves a direct
1465+// session kill whose teardown snapshot a concurrent token resume invalidated
1466+// reports a clear retryable outcome instead of a permanent delete failure, and
1467+// leaves the session live.
1468+func TestHandleKillSessionParticipantsChangedReportsRetryable(t *testing.T) {
1469+ d, sess, ac, _ := newManualSessionWithPTYs(t, newQuietPTY())
1470+
1471+ d.afterAttachmentEffectParticipantsSnapshotted = func(action string, _ []*attachedClient) {
1472+ if action != "" {
1473+ return
1474+ }
1475+ d.afterAttachmentEffectParticipantsSnapshotted = nil
1476+ // A token resume replaces the exact transport after the participant
1477+ // snapshot was taken, so the kill aborts before removing anything.
1478+ fresh := &closeTrackingTransport{}
1479+ ac.replaceTransport(fresh)
1480+ ac.installTestAttachmentCapability(sess.captureAttachmentCapability(ac, fresh))
1481+ }
1482+ t.Cleanup(func() { d.afterAttachmentEffectParticipantsSnapshotted = nil })
1483+
1484+ tr, sends := newCapturingTransport(t)
1485+ d.handleKill(tr, protocol.Kill{Scope: protocol.KillSession, Name: "work"})
1486+
1487+ frame := awaitFrame(t, sends, "Error")
1488+ msg, ok := decodeServerMessage(t, frame).(protocol.ErrorMsg)
1489+ require.True(t, ok)
1490+ require.Equal(t, protocol.ErrInternal, msg.Code)
1491+ require.Equal(t, sessionKillRetryMessage, msg.Text)
1492+ require.Equal(t, 1, sessionCount(d), "an aborted direct kill must leave the session live")
1493+ require.Same(t, sess, firstSession(d))
1494+}
1495+
1496+// TestCloseTabParticipantsChangedReportsRetryable proves a last-tab close whose
1497+// teardown snapshot a concurrent token resume invalidated reports a clear
1498+// retryable outcome to both the command result and the attached client, and
1499+// leaves the session and its tab live.
1500+func TestCloseTabParticipantsChangedReportsRetryable(t *testing.T) {
1501+ d, sess, ac, _ := newManualSessionWithPTYs(t, newQuietPTY())
1502+
1503+ d.afterAttachmentEffectParticipantsSnapshotted = func(action string, _ []*attachedClient) {
1504+ if action != "" {
1505+ return
1506+ }
1507+ d.afterAttachmentEffectParticipantsSnapshotted = nil
1508+ fresh := &closeTrackingTransport{}
1509+ ac.replaceTransport(fresh)
1510+ ac.installTestAttachmentCapability(sess.captureAttachmentCapability(ac, fresh))
1511+ }
1512+ t.Cleanup(func() { d.afterAttachmentEffectParticipantsSnapshotted = nil })
1513+
1514+ result := sendCommand(t, d, protocol.CommandRequest{Slug: "close-tab", TargetSession: "work"})
1515+ require.False(t, result.OK)
1516+ require.Equal(t, protocol.ErrInternal, result.Code)
1517+ require.Equal(t, sessionKillRetryMessage, result.Text)
1518+
1519+ toasts := awaitToastCount(t, ac, 1)
1520+ require.Contains(t, toasts[0].Message, "couldn't close tab")
1521+ require.Contains(t, toasts[0].Message, "try again")
1522+
1523+ sess.mu.Lock()
1524+ tabs := len(sess.tabs)
1525+ sess.mu.Unlock()
1526+ require.Equal(t, 1, tabs, "an aborted last-tab close must retain the tab")
1527+ require.Equal(t, 1, sessionCount(d), "an aborted last-tab close must retain the session")
1528+}
M internal/usecase/daemon/daemon.go +128 -63
@@ -91,11 +91,13 @@ type Daemon struct {
9191 lastAllocatedCreatedAt int64
9292 mruSeq atomic.Uint64
9393 creationRequestSeq atomic.Uint64
94- // closing marks that shutdown has irreversibly begun. It is set under mu,
95- // atomically with the event that makes shutdown inevitable (the registry
96- // emptying in killSession, or shutdownAll starting), and checked by route
97- // under the same mutex — so a Hello racing shutdown can never insert a new
98- // session that nobody would tear down.
94+ // closing marks that explicit daemon shutdown (KillDaemon/process
95+ // cancellation) has irreversibly begun. It is set under mu when
96+ // shutdownAll starts, and checked by route under the same mutex — so a Hello
97+ // racing shutdown can never insert a new session that nobody would tear
98+ // down. Neither an empty registry (final session removal) nor a KillAll
99+ // purge sets it: the daemon survives empty→occupied→empty and survives a
100+ // purge, and only an explicit stop/process cancellation ends it.
99101 closing bool
100102
101103 // moveLifecycleMu is the daemon-level admission gate for transferable
@@ -110,6 +112,28 @@ type Daemon struct {
110112 paneProcessCtx context.Context
111113 paneProcessCancel context.CancelFunc
112114
115+ // purgeAllMu grants one daemon-wide purge (KillAll) owner at a time so two
116+ // concurrent kill-all requests never interleave their exact lifecycle sets.
117+ // It is not an admission gate; see purgeAdmissionClosing below.
118+ purgeAllMu sync.Mutex
119+ // purgeAdmissionClosing is the transient KillAll admission gate. Unlike
120+ // closing and moveLifecycleClosing it is re-openable: purgeAllSessions sets
121+ // it under mu, removes exactly the live and stopped/broken/durable records
122+ // captured at that instant, then clears it so the daemon keeps serving.
123+ // Creation and restoration wait on the changed channel (a blocked writer
124+ // keeps new admissions out), while moves and internal creations reject, so
125+ // no transition can be inserted into the purge's exact lifecycle set.
126+ // purgeAdmissionActive counts admitted transitions the purge must drain.
127+ // purgeEpoch increments once at each purge that acquires the gate (after the
128+ // drain) so a restoration worker blocked on admission can detect a purge
129+ // that completed while it waited and must not resurrect the records captured
130+ // before that purge.
131+ // All three fields are guarded by mu.
132+ purgeAdmissionClosing bool
133+ purgeAdmissionActive uint
134+ purgeAdmissionChanged chan struct{}
135+ purgeEpoch uint64
136+
113137 // notifies holds one completion channel per in-flight async Detached
114138 // notification (guarded by mu, pruned on insert). Channels rather than a
115139 // WaitGroup: notifications are spawned from arbitrary goroutines while
@@ -180,6 +204,21 @@ type Daemon struct {
180204 // afterAttachmentEffectsFrozen observes the lock-free boundary after all affected
181205 // attachment gates are frozen and drained, before architecture publication.
182206 afterAttachmentEffectsFrozen func()
207+ // afterPurgeAdmission is a deterministic test seam that runs after a KillAll
208+ // purge has closed purge admission and taken the closing fence, while it
209+ // still owns purgeAllMu. It lets tests observe purge serialization.
210+ afterPurgeAdmission func()
211+ // beforePurgeLiveSessionKill is a deterministic test seam that runs after a
212+ // KillAll purge's per-unit shutdown precheck and before that live unit
213+ // attempts teardown ownership. Tests use it to let an explicit stop win that
214+ // exact window, proving a stop-owned unit is never reported as a success.
215+ beforePurgeLiveSessionKill func(*session)
216+ // afterSessionKillQuarantine is a deterministic test seam that runs while a
217+ // teardown owns the session and its snapshot coordinator is quarantined, but
218+ // before the pre-publication revalidation. Tests use it to change the
219+ // participant set in that window and prove a post-quarantine abort rolls the
220+ // quarantine back so a surviving session keeps checkpointing.
221+ afterSessionKillQuarantine func(*session)
183222 // beforeAttachmentTransitionIdentityAdmission is a deterministic test seam
184223 // after a ready transition published its capability and before the committed
185224 // route identity is admitted, so tests can supersede that exact capability in
@@ -348,6 +387,10 @@ type Daemon struct {
348387 hardCtx context.Context
349388 hardCancel context.CancelFunc
350389
390+ // done is closed exactly once by an explicit daemon stop (shutdownAll) to
391+ // break Serve's accept loop. It is deliberately not closed by final session
392+ // removal or by a KillAll purge: an empty registry and a purged registry
393+ // both keep the daemon serving.
351394 done chan struct{}
352395 doneOnce sync.Once
353396
@@ -447,6 +490,10 @@ func WithRuntimeObserver(observer ports.SerializedRuntimeObserver) Option {
447490 // WithRemoteMonitor installs the snapshot-only remote directory and the
448491 // monitor runner. Serve starts the runner without waiting for registry,
449492 // cache or runtime readiness and stops it with its own context.
493+//
494+// The daemon-side monitor stays intentionally active in this state: it is part
495+// of the coordinated P7 removal set, so no polling gate, feature switch, or
496+// early shutdown path is added here (GO-001, deferred to P7).
450497 func WithRemoteMonitor(directory ports.RemoteDirectory, run func(context.Context) error) Option {
451498 return func(d *Daemon) {
452499 d.remoteDirectory = directory
@@ -792,8 +839,12 @@ func New(ptys ports.PTYFactory, clock ports.Clock, log *slog.Logger, opts ...Opt
792839 const remoteMonitorShutdownJoin = 3 * time.Second
793840
794841 // Serve runs the accept loop over l, owning it for the loop's lifetime. It
795-// returns when the last session is removed or ctx is cancelled; on the latter
796-// path attached clients are detached with ReasonServerShutdown.
842+// returns only on ctx cancellation or an explicit daemon shutdown request
843+// (KillDaemon), never when the session registry drains to empty: final
844+// session removal is independent of daemon shutdown so the daemon survives
845+// empty→occupied→empty and can accept fresh sessions again. A KillAll purge
846+// leaves Serve running too. On the ctx path
847+// attached clients are detached with ReasonServerShutdown.
797848 func (d *Daemon) Serve(ctx context.Context, l ports.ServerListener) error {
798849 d.serveCtx, d.serveCancel = context.WithCancel(ctx)
799850 defer d.serveCancel()
@@ -866,8 +917,9 @@ func (d *Daemon) Serve(ctx context.Context, l ports.ServerListener) error {
866917 } else {
867918 d.closeRestoreDone()
868919 }
869- // Break the accept loop when either the parent context is cancelled or the
870- // registry drains to empty: both close the listener, which fails Accept.
920+ // Break the accept loop only on an explicit shutdown: the parent context
921+ // being cancelled or an explicit daemon stop request (d.done). An empty
922+ // registry never closes the listener, so the daemon keeps accepting.
871923 go func() {
872924 select {
873925 case <-d.serveCtx.Done():
@@ -908,7 +960,7 @@ func (d *Daemon) Serve(ctx context.Context, l ports.ServerListener) error {
908960 snapshotDeadline = newSnapshotShutdownDeadline(d.clock)
909961 defer snapshotDeadline.stop()
910962 }
911- d.terminateAllWithSnapshotDeadline(protocol.ReasonServerShutdown, false, snapshotDeadline)
963+ d.terminateAllForShutdown(protocol.ReasonServerShutdown, snapshotDeadline)
912964 d.waitNotifies()
913965 d.hardCancel()
914966 d.serveCancel()
@@ -928,7 +980,7 @@ func (d *Daemon) Serve(ctx context.Context, l ports.ServerListener) error {
928980 d.persistShutdownSnapshotFailure(name, context.DeadlineExceeded)
929981 }
930982 d.WaitDurableWriters()
931- d.terminateAllWithSnapshotDeadline(protocol.ReasonServerShutdown, false, snapshotDeadline)
983+ d.terminateAllForShutdown(protocol.ReasonServerShutdown, snapshotDeadline)
932984 d.waitSessionWorkersWithSnapshotDeadline(snapshotDeadline)
933985 d.waitNotifies()
934986 if err := d.flushCatalogue(); err != nil {
@@ -942,58 +994,41 @@ func (d *Daemon) Serve(ctx context.Context, l ports.ServerListener) error {
942994 return nil
943995 }
944996
945-// shutdownAll marks the daemon closing and kills every live session. Setting
946-// closing under the same lock as the snapshot guarantees no session can be
947-// inserted after the snapshot: route rejects once closing is set, and both run
948-// under d.mu. killSession (which relocks) runs after the lock is released.
997+// shutdownAll is the explicit daemon-stop operation (wire KillDaemon). It is
998+// the only caller that irreversibly ends the daemon: it closes move admission,
999+// marks closing, cancels the daemon-wide pane process context, preserves every
1000+// live session as stopped durable authority, and closes done so Serve returns.
1001+// KillAll never reaches this path. Setting closing under the same lock as the
1002+// snapshot guarantees no session can be inserted after the snapshot: route
1003+// rejects once closing is set, and both run under d.mu. killSession (which
1004+// relocks) runs after the lock is released.
9491005 func (d *Daemon) shutdownAll(reason uint8) (checkpointIncomplete bool) {
950- return d.terminateAllWithSnapshotDeadline(reason, false, nil)
951-}
952-
953-func (d *Daemon) purgeAll(reason uint8) (checkpointIncomplete bool) {
954- return d.terminateAllWithSnapshotDeadline(reason, true, nil)
1006+ return d.terminateAllForShutdown(reason, nil)
9551007 }
9561008
957-func (d *Daemon) terminateAllWithSnapshotDeadline(reason uint8, purge bool, deadline *snapshotShutdownDeadline) (checkpointIncomplete bool) {
1009+// terminateAllForShutdown performs the irreversible daemon teardown shared by
1010+// shutdownAll and Serve's exit path. It is separate from purgeAllSessions,
1011+// which leaves every daemon-wide shutdown fact untouched.
1012+func (d *Daemon) terminateAllForShutdown(reason uint8, deadline *snapshotShutdownDeadline) (checkpointIncomplete bool) {
9581013 d.closeMoveLifecycles()
9591014 d.mu.Lock()
9601015 d.closing = true
1016+ // Wake any KillAll waiting on purge admission so it defers to this stop
1017+ // instead of holding the gate until its bounded drain deadline expires.
1018+ d.signalPurgeAdmissionChangedLocked()
9611019 d.purgeAllParkingLocked()
9621020 parkedRetirements := d.purgeAllParkedLocked()
9631021 d.purgeAllSuspendedLocked()
9641022 snapshot := d.sessionsSnapshotLocked()
965- stoppedNames := make([]string, 0, len(d.inactive))
966- if purge {
967- for name := range d.inactive {
968- stoppedNames = append(stoppedNames, name)
969- }
970- sort.Strings(stoppedNames)
971- }
972- empty := len(snapshot) == 0
9731023 d.mu.Unlock()
9741024 d.finishParkedAttachmentRetirements(parkedRetirements)
975- if !purge {
976- // Publish preservation policy for the complete registry snapshot before
977- // cancelling any PTY. Cancellation-driven EOF is allowed to own teardown,
978- // but its disposition must remain daemon shutdown rather than session purge.
979- for _, s := range snapshot {
980- s.reserveShutdownTeardown()
981- }
982- }
983- d.log.Info("session termination begin", "reason", reason, "purge", purge, "live_sessions", len(snapshot), "stopped_sessions", len(stoppedNames))
984- // Purge stopped authority before signaling Serve to exit. Once the final
985- // live registry entry closes, Serve cancels its context; deleting stopped
986- // records afterward would be interrupted and retain broken catalogue entries.
987- for _, name := range stoppedNames {
988- if err := d.retryStoppedPurge(name); err != nil {
989- checkpointIncomplete = true
990- d.log.Error("deleting stopped session during kill all failed", "err", err, "session", name)
991- }
992- }
993- if empty {
994- d.doneOnce.Do(func() { close(d.done) })
995- return checkpointIncomplete
1025+ // Publish preservation policy for the complete registry snapshot before
1026+ // cancelling any PTY. Cancellation-driven EOF is allowed to own teardown,
1027+ // but its disposition must remain daemon shutdown rather than session purge.
1028+ for _, s := range snapshot {
1029+ s.reserveShutdownTeardown()
9961030 }
1031+ d.log.Info("session termination begin", "reason", reason, "live_sessions", len(snapshot))
9971032 for _, s := range snapshot {
9981033 // Cancellation and PTY closure must not wait behind a teardown owner that
9991034 // is blocked in snapshot publication or purge work.
@@ -1002,11 +1037,11 @@ func (d *Daemon) terminateAllWithSnapshotDeadline(reason uint8, purge bool, dead
10021037 ephemeral := s.ephemeral
10031038 s.mu.Unlock()
10041039 s.stopInMemoryLifecycle()
1005- if err := d.killSessionWithSnapshotDeadline(s, reason, purge, deadline, nil); err != nil {
1040+ if err := d.killSessionWithSnapshotDeadline(s, reason, false, deadline, nil); err != nil {
10061041 checkpointIncomplete = true
10071042 d.log.Error("closing session with unpersisted terminal state", "err", err)
10081043 }
1009- if !purge && !ephemeral && deadline != nil {
1044+ if !ephemeral && deadline != nil {
10101045 select {
10111046 case <-deadline.Done():
10121047 checkpointIncomplete = true
@@ -1291,15 +1326,21 @@ func (d *Daemon) handleList(tr ports.ServerConnection) {
12911326 }
12921327
12931328 // handleKill terminates the requested live session or stopped named session,
1294-// or all sessions, and closes the control connection; the resulting EOF is the
1295-// client's success signal.
1329+// purges every session (KillAll), or stops the daemon (KillDaemon), and closes
1330+// the control connection; the resulting EOF is the client's success signal.
1331+// KillAll leaves the daemon active; KillDaemon is the explicit stop.
12961332 func (d *Daemon) handleKill(tr ports.ServerConnection, request protocol.Kill) {
12971333 defer func() { _ = tr.Close() }()
12981334
12991335 switch request.Scope {
13001336 case protocol.KillAll:
1301- if d.purgeAll(protocol.ReasonSessionKilled) {
1302- _ = d.boundedControlSend(tr, serverError(protocol.ErrInternal, "one or more sessions could not be deleted"))
1337+ result := d.purgeAllSessions(protocol.ReasonSessionKilled)
1338+ if result.Superseded() {
1339+ d.logControlSendFailure("kill all superseded response", d.boundedControlSend(tr, serverError(protocol.ErrServerShutdown, "daemon is shutting down")))
1340+ return
1341+ }
1342+ if result.Failed() {
1343+ d.logControlSendFailure("kill all failure response", d.boundedControlSend(tr, serverError(protocol.ErrInternal, result.summary())))
13031344 }
13041345 return
13051346 case protocol.KillDaemon:
@@ -1307,7 +1348,7 @@ func (d *Daemon) handleKill(tr ports.ServerConnection, request protocol.Kill) {
13071348 return
13081349 case protocol.KillSession:
13091350 default:
1310- _ = d.boundedControlSend(tr, serverError(protocol.ErrInternal, "invalid kill scope"))
1351+ d.logControlSendFailure("invalid kill scope response", d.boundedControlSend(tr, serverError(protocol.ErrInternal, "invalid kill scope")))
13111352 return
13121353 }
13131354
@@ -1320,7 +1361,7 @@ func (d *Daemon) handleKill(tr ports.ServerConnection, request protocol.Kill) {
13201361 // deletion order as live and offline purges.
13211362 if err := d.retryStoppedPurge(request.Name); err != nil {
13221363 d.log.Warn("deleting stopped session failed", "err", err, "session", request.Name)
1323- _ = d.boundedControlSend(tr, serverError(protocol.ErrInternal, "deleting stopped session failed"))
1364+ d.logControlSendFailure("stopped delete failure response", d.boundedControlSend(tr, serverError(protocol.ErrInternal, "deleting stopped session failed")))
13241365 }
13251366 return
13261367 }
@@ -1328,11 +1369,18 @@ func (d *Daemon) handleKill(tr ports.ServerConnection, request protocol.Kill) {
13281369 d.mu.Unlock()
13291370
13301371 if target == nil {
1331- _ = d.boundedControlSend(tr, serverError(protocol.ErrNoSuchSession, "no such session: "+request.Name))
1372+ d.logControlSendFailure("no such session response", d.boundedControlSend(tr, serverError(protocol.ErrNoSuchSession, "no such session: "+request.Name)))
13321373 return
13331374 }
13341375 if err := d.killSession(target, protocol.ReasonSessionKilled, true); err != nil {
1335- _ = d.boundedControlSend(tr, serverError(protocol.ErrInternal, "deleting persisted session failed"))
1376+ if errors.Is(err, errSessionKillParticipantsChanged) {
1377+ // The exact session was not removed; a concurrent detach or token
1378+ // resume invalidated the teardown snapshot. Report the retryable
1379+ // outcome instead of a permanent delete failure.
1380+ d.logControlSendFailure("session delete retry response", d.boundedControlSend(tr, serverError(protocol.ErrInternal, sessionKillRetryMessage)))
1381+ return
1382+ }
1383+ d.logControlSendFailure("session delete failure response", d.boundedControlSend(tr, serverError(protocol.ErrInternal, "deleting persisted session failed")))
13361384 }
13371385 }
13381386
@@ -1676,7 +1724,15 @@ func (d *Daemon) routeWithContext(ctx context.Context, h protocol.Hello, tr port
16761724 if h.ExactTarget != nil && h.ResumeToken == 0 && h.Name != h.ExactTarget.SessionName {
16771725 return nil, nil, &protoErr{protocol.ErrNoSuchSession, "exact session target name mismatch"}
16781726 }
1727+ // A remote exact-target handoff is an admitted transition like a local
1728+ // create/restore: it reserves purge admission before dispatch, so a Hello
1729+ // arriving during a KillAll waits on the gate exactly as a local route does
1730+ // instead of being rejected outright.
16791731 if h.RemoteTarget != nil && h.ResumeToken == 0 {
1732+ if err := d.acquirePurgeAdmission(ctx); err != nil {
1733+ return nil, nil, err
1734+ }
1735+ defer d.releasePurgeAdmission()
16801736 return d.routeRemoteTargetWithContext(ctx, h, tr)
16811737 }
16821738 if h.ExactTarget != nil && h.ResumeToken == 0 {
@@ -1731,11 +1787,20 @@ func (d *Daemon) routeWithContext(ctx context.Context, h protocol.Hello, tr port
17311787 h.Intent == protocol.IntentEphemeral {
17321788 return nil, nil, &protoErr{protocol.ErrNoSuchTarget, "daemon-owned environment requires an exact remote target"}
17331789 }
1790+ // KillAll admission gate: a purge owns the exact lifecycle set it captured,
1791+ // so a Hello arriving during it waits here and is then admitted as a create
1792+ // or restore after the purge, never inserted into the set being removed.
1793+ // This is transient and re-opened by the purge; it is not daemon closing.
1794+ if err := d.acquirePurgeAdmission(ctx); err != nil {
1795+ return nil, nil, err
1796+ }
1797+ defer d.releasePurgeAdmission()
17341798 d.mu.Lock()
1735- // Shutdown/create interlock: once shutdown has begun (last session removed,
1736- // or shutdownAll started) no new session may be created and no attach may
1799+ // Shutdown/create interlock: once an explicit shutdown has begun
1800+ // (shutdownAll started) no new session may be created and no attach may
17371801 // proceed — the teardown snapshot has already been (or is being) taken, so
1738- // anything inserted now would leak its PTY and hang Serve.
1802+ // anything inserted now would leak its PTY and hang Serve. An empty registry
1803+ // alone is not shutdown, and a KillAll purge is gated by admission above.
17391804 if d.closing {
17401805 d.mu.Unlock()
17411806 return nil, nil, &protoErr{protocol.ErrServerShutdown, "daemon is shutting down"}
M internal/usecase/daemon/durable_recovery_test.go +18 -18
@@ -275,10 +275,10 @@ func TestFinalCheckpointFailurePreservesCatalogue(t *testing.T) {
275275
276276 func TestCatalogueRestoreIndependent(t *testing.T) {
277277 t.Run("empty catalogue restoration", func(t *testing.T) {
278- emptyDaemon, emptyCatalogue := newDurableRecoveryDaemon(t, nil, &durableRecoveryRepository{})
278+ emptyDaemon, _ := newDurableRecoveryDaemon(t, nil, &durableRecoveryRepository{})
279279 emptyDone := make(chan struct{})
280280 go func() {
281- emptyDaemon.restoreCatalogue(context.Background(), mustDurableRecords(t, emptyCatalogue))
281+ emptyDaemon.restoreIncrementalSnapshots(context.Background())
282282 close(emptyDone)
283283 }()
284284 select {
@@ -299,7 +299,7 @@ func TestCatalogueRestoreIndependent(t *testing.T) {
299299 }
300300 d, catalogue := newDurableRecoveryDaemon(t, []domain.CatalogueRecord{healthy, broken}, repository)
301301
302- d.restoreCatalogue(context.Background(), mustDurableRecords(t, catalogue))
302+ d.restoreIncrementalSnapshots(context.Background())
303303
304304 d.mu.Lock()
305305 healthyEntry := d.inactive[healthy.Name]
@@ -335,7 +335,7 @@ func TestCatalogueRestoreResetsOnlyIncompatibleCheckpoint(t *testing.T) {
335335 }
336336 d, catalogue := newDurableRecoveryDaemon(t, []domain.CatalogueRecord{incompatible, healthy}, repository)
337337
338- d.restoreCatalogue(context.Background(), mustDurableRecords(t, catalogue))
338+ d.restoreIncrementalSnapshots(context.Background())
339339
340340 fresh, ok, err := catalogue.Record(incompatible.Name)
341341 require.NoError(t, err)
@@ -370,7 +370,7 @@ func TestIncompatibleCheckpointDeleteFailurePublishesFreshStoppedAuthority(t *te
370370 restoreDone := d.inactive[record.Name].restoreDone
371371 d.mu.Unlock()
372372
373- restoreErr := d.restoreRecord(context.Background(), record)
373+ restoreErr := d.restoreRecord(context.Background(), record, d.purgeEpochSnapshot())
374374 require.NoError(t, restoreErr, "post-commit cleanup failure must not fail the authority transition")
375375
376376 fresh, ok, err := catalogue.Record(record.Name)
@@ -416,7 +416,7 @@ func TestIncompatibleCheckpointPreCommitFailureReturnsError(t *testing.T) {
416416 d, catalogue := newDurableRecoveryDaemon(t, []domain.CatalogueRecord{record}, repository)
417417 catalogue.replaceErr = cause
418418
419- err := d.restoreRecord(context.Background(), record)
419+ err := d.restoreRecord(context.Background(), record, d.purgeEpochSnapshot())
420420
421421 require.ErrorIs(t, err, cause)
422422 persisted, ok, recordErr := catalogue.Record(record.Name)
@@ -445,7 +445,7 @@ func TestRestoreLoadFailuresRetainCheckpointWithoutDeletion(t *testing.T) {
445445 }
446446 d, catalogue := newDurableRecoveryDaemon(t, []domain.CatalogueRecord{record}, repository)
447447
448- d.restoreCatalogue(context.Background(), mustDurableRecords(t, catalogue))
448+ d.restoreIncrementalSnapshots(context.Background())
449449
450450 persisted, ok, err := catalogue.Record(record.Name)
451451 require.NoError(t, err)
@@ -467,9 +467,9 @@ func TestCatalogueRestoreAggregateOver4096(t *testing.T) {
467467 records[i] = durableRecoveryRecord(i)
468468 repository.generations[records[i].Name] = validGeneration(t, records[i])
469469 }
470- d, catalogue := newDurableRecoveryDaemon(t, records, repository)
470+ d, _ := newDurableRecoveryDaemon(t, records, repository)
471471
472- d.restoreCatalogue(context.Background(), mustDurableRecords(t, catalogue))
472+ d.restoreIncrementalSnapshots(context.Background())
473473
474474 require.Len(t, d.inactive, count)
475475 for _, record := range records {
@@ -481,11 +481,11 @@ func TestCatalogueRestoreAggregateOver4096(t *testing.T) {
481481 func TestCatalogueRestoreSingleSessionOver4096(t *testing.T) {
482482 record := durableRecoveryRecord(0)
483483 repository := &durableRecoveryRepository{generations: map[string]ports.SnapshotGeneration{record.Name: validGeneration(t, record)}, errors: make(map[string]error), loads: make(map[string]int), repairs: make(map[string]domain.CheckpointRef)}
484- d, catalogue := newDurableRecoveryDaemon(t, []domain.CatalogueRecord{record}, repository)
484+ d, _ := newDurableRecoveryDaemon(t, []domain.CatalogueRecord{record}, repository)
485485
486486 // The repository may contain any number of stale entries; restoration has no
487487 // discovery API and therefore performs exactly one direct catalogue lookup.
488- d.restoreCatalogue(context.Background(), mustDurableRecords(t, catalogue))
488+ d.restoreIncrementalSnapshots(context.Background())
489489
490490 require.Equal(t, protocol.SessionDown, d.inactive[record.Name].state)
491491 require.Equal(t, 1, repository.loads[record.Name])
@@ -498,7 +498,7 @@ func TestCatalogueRestoreIncarnationMismatch(t *testing.T) {
498498 repository := &durableRecoveryRepository{generations: map[string]ports.SnapshotGeneration{record.Name: generation}, errors: make(map[string]error), loads: make(map[string]int), repairs: make(map[string]domain.CheckpointRef)}
499499 d, catalogue := newDurableRecoveryDaemon(t, []domain.CatalogueRecord{record}, repository)
500500
501- d.restoreCatalogue(context.Background(), mustDurableRecords(t, catalogue))
501+ d.restoreIncrementalSnapshots(context.Background())
502502
503503 require.Equal(t, protocol.SessionBroken, d.inactive[record.Name].state)
504504 got, _, _ := catalogue.Record(record.Name)
@@ -674,7 +674,7 @@ func TestPersistAndRegisterRestoredSessionRegistrationFailurePreservesStoppedEnt
674674 sess := newSnapshotTestSession(t, record.Name, false, record.Cwd)
675675 sess.incarnation = record.IncarnationID
676676
677- registered, err := d.persistAndRegisterRestoredSession(t.Context(), sess)
677+ registered, err := d.persistAndRegisterRestoredSession(t.Context(), sess, d.purgeEpochSnapshot())
678678
679679 require.ErrorContains(t, err, "registry rejected")
680680 require.False(t, registered)
@@ -744,7 +744,7 @@ func TestPersistAndRegisterRestoredSessionRejectsReplacementIncarnation(t *testi
744744 sess := newSnapshotTestSession(t, record.Name, false, record.Cwd)
745745 sess.incarnation = record.IncarnationID
746746
747- registered, err := d.persistAndRegisterRestoredSession(t.Context(), sess)
747+ registered, err := d.persistAndRegisterRestoredSession(t.Context(), sess, d.purgeEpochSnapshot())
748748 require.ErrorContains(t, err, "incarnation")
749749 require.False(t, registered)
750750 require.Empty(t, d.sessions)
@@ -842,7 +842,7 @@ func TestAttachBarrierSurvivesLiveRegistryPublication(t *testing.T) {
842842 d.mu.Unlock()
843843 sess := &session{sessionCore: sessionCore{name: record.Name, incarnation: record.IncarnationID}}
844844
845- registered, err := d.persistAndRegisterRestoredSession(t.Context(), sess)
845+ registered, err := d.persistAndRegisterRestoredSession(t.Context(), sess, d.purgeEpochSnapshot())
846846 require.NoError(t, err)
847847 require.True(t, registered)
848848
@@ -879,7 +879,7 @@ func TestRestoreCancellationTransitionsBeforeAttachCompletion(t *testing.T) {
879879 restoreCtx, cancelRestore := context.WithCancel(context.Background())
880880 restored := make(chan struct{})
881881 go func() {
882- d.restoreCatalogue(restoreCtx, mustDurableRecords(t, catalogue))
882+ d.restoreIncrementalSnapshots(restoreCtx)
883883 close(restored)
884884 }()
885885 <-repository.started
@@ -990,9 +990,9 @@ func TestCatalogueRestoreDegradedVisible(t *testing.T) {
990990 record := durableRecoveryRecord(0)
991991 record.DegradedReason = "checkpoint validation failed"
992992 repository := &durableRecoveryRepository{generations: make(map[string]ports.SnapshotGeneration), errors: make(map[string]error), loads: make(map[string]int), repairs: make(map[string]domain.CheckpointRef)}
993- d, catalogue := newDurableRecoveryDaemon(t, []domain.CatalogueRecord{record}, repository)
993+ d, _ := newDurableRecoveryDaemon(t, []domain.CatalogueRecord{record}, repository)
994994
995- d.restoreCatalogue(context.Background(), mustDurableRecords(t, catalogue))
995+ d.restoreIncrementalSnapshots(context.Background())
996996
997997 require.Equal(t, protocol.SessionBroken, d.inactive[record.Name].state)
998998 require.Zero(t, repository.loads[record.Name])
M internal/usecase/daemon/move_lifecycle.go +29 -4
@@ -33,11 +33,27 @@ func (d *Daemon) reserveMoveLifecycles(source, destination *session) (*moveLifec
3333 }
3434 d.mu.Lock()
3535 closing := d.closing
36+ purgeClosing := d.purgeAdmissionClosing
37+ if !closing && !purgeClosing {
38+ // A move transfers live state across the exact lifecycle set a KillAll
39+ // purge captured, so it reserves the purge admission gate too: the purge
40+ // drains this reservation before it snapshots, and a move arriving during
41+ // the purge is rejected instead of racing the set.
42+ d.purgeAdmissionActive++
43+ }
3644 d.mu.Unlock()
3745 if closing {
3846 d.moveLifecycleMu.Unlock()
3947 return nil, errMoveLifecycleUnavailable
4048 }
49+ if purgeClosing {
50+ // A transient KillAll purge owns the exact lifecycle set, so no move can
51+ // reserve it. Unlike a shutdown this is retry state, so report the same
52+ // typed admission outcome as the other command paths instead of a
53+ // permanent "target gone" verdict.
54+ d.moveLifecycleMu.Unlock()
55+ return nil, errPurgeAdmissionClosed
56+ }
4157 d.moveLifecycleActive++
4258 d.signalMoveLifecycleChangedLocked()
4359 d.moveLifecycleMu.Unlock()
@@ -53,8 +69,7 @@ func (d *Daemon) reserveMoveLifecycles(source, destination *session) (*moveLifec
5369 if sess.teardownActive {
5470 unlockMoveSessions(sessions)
5571 d.moveLifecycleMu.Lock()
56- d.moveLifecycleActive--
57- d.signalMoveLifecycleChangedLocked()
72+ d.releaseMoveAdmissionLocked()
5873 d.moveLifecycleMu.Unlock()
5974 return nil, errMoveLifecycleUnavailable
6075 }
@@ -99,8 +114,7 @@ func (r *moveLifecycleReservation) Release() {
99114 sess.signalTeardownChangedLocked()
100115 }
101116 unlockMoveSessions(r.sessions)
102- r.daemon.moveLifecycleActive--
103- r.daemon.signalMoveLifecycleChangedLocked()
117+ r.daemon.releaseMoveAdmissionLocked()
104118 r.daemon.moveLifecycleMu.Unlock()
105119 })
106120 }
@@ -139,6 +153,17 @@ func (d *Daemon) closeMoveLifecycles() {
139153 }
140154 }
141155
156+// releaseMoveAdmissionLocked drops one move's move-gate and KillAll purge-gate
157+// reservations. Caller holds moveLifecycleMu; the purge gate is guarded by mu.
158+func (d *Daemon) releaseMoveAdmissionLocked() {
159+ d.moveLifecycleActive--
160+ d.signalMoveLifecycleChangedLocked()
161+ d.mu.Lock()
162+ d.purgeAdmissionActive--
163+ d.signalPurgeAdmissionChangedLocked()
164+ d.mu.Unlock()
165+}
166+
142167 func (d *Daemon) moveLifecycleChangeLocked() chan struct{} {
143168 if d.moveLifecycleChanged == nil {
144169 d.moveLifecycleChanged = make(chan struct{})
M internal/usecase/daemon/move_lifecycle_test.go +20 -0
@@ -147,6 +147,26 @@ func TestMoveLifecycleReservationRejectsTeardownWithoutLeakingCounts(t *testing.
147147 }
148148 }
149149
150+// TestMoveLifecycleReservationRejectedByPurgeAdmissionIsRetryable proves a
151+// move that arrives while a KillAll purge owns the exact lifecycle set is
152+// rejected with the transient purge-admission sentinel rather than the
153+// permanent shutdown verdict, and that the rejection leaks no move or purge
154+// admission counts.
155+func TestMoveLifecycleReservationRejectedByPurgeAdmissionIsRetryable(t *testing.T) {
156+ d := newTestDaemon(t, nil, stubClock{})
157+ source := &session{sessionCore: sessionCore{id: "source"}}
158+ destination := &session{sessionCore: sessionCore{id: "destination"}}
159+
160+ beginTestPurgeAdmission(t, d)
161+ reservation, err := d.reserveMoveLifecycles(source, destination)
162+ require.ErrorIs(t, err, errPurgeAdmissionClosed)
163+ require.NotErrorIs(t, err, errMoveLifecycleUnavailable)
164+ require.Nil(t, reservation)
165+ awaitDaemonMoveActive(t, d, 0)
166+ d.endPurgeAdmission()
167+ requirePurgeAdmissionBalanced(t, d)
168+}
169+
150170 func TestMoveLifecycleReservationDeduplicatesAndReleasesExactlyOnce(t *testing.T) {
151171 d := newTestDaemon(t, nil, stubClock{})
152172 sess := &session{sessionCore: sessionCore{id: "same"}}
M internal/usecase/daemon/move_rejection.go +11 -0
@@ -114,6 +114,11 @@ func movePickerUserError(err error) error {
114114 if errors.As(err, &userErr) {
115115 return err
116116 }
117+ if errors.Is(err, errPurgeAdmissionClosed) {
118+ // A transient KillAll admission rejection is retry state: surface a
119+ // retryable warning instead of a permanent "Move failed" verdict.
120+ return domain.UserWarn(domain.NoticeSessionUnavailable, sessionKillRetryMessage, err)
121+ }
117122 descriptor, ok := moveRejectionDescriptorFor(err)
118123 if !ok {
119124 descriptor = moveRejectionDescriptorByReason(moveRejectionInvalid)
@@ -145,6 +150,12 @@ func normalizeMoveRejection(err error) error {
145150 if errors.As(err, &rejection) {
146151 return err
147152 }
153+ if errors.Is(err, errPurgeAdmissionClosed) {
154+ // A transient KillAll admission rejection is retry state, not a move
155+ // rejection: preserve the typed sentinel so the command and palette
156+ // adapters map it consistently with the other command paths.
157+ return err
158+ }
148159 if errors.Is(err, layout.ErrTooSmall) {
149160 return errMoveTooSmall
150161 }
M internal/usecase/daemon/move_rejection_test.go +20 -0
@@ -50,6 +50,26 @@ func TestNormalizeMoveRejectionDoesNotLeakInternalErrors(t *testing.T) {
5050 require.ErrorIs(t, got, errMovePaneInvalid)
5151 }
5252
53+// TestMoveRejectionPreservesTransientPurgeAdmission proves a transient KillAll
54+// admission rejection is not wrapped as a move rejection or leaked as an
55+// internal detail: the command adapter keeps the typed sentinel it maps
56+// retryably, and the palette adapter renders a retryable warning.
57+func TestMoveRejectionPreservesTransientPurgeAdmission(t *testing.T) {
58+ normalized := normalizeMoveRejection(errPurgeAdmissionClosed)
59+ require.ErrorIs(t, normalized, errPurgeAdmissionClosed)
60+ require.NotErrorIs(t, normalized, errMovePaneInvalid)
61+ var rejection *moveRejection
62+ require.False(t, errors.As(normalized, &rejection))
63+
64+ paletteErr := movePickerUserError(errPurgeAdmissionClosed)
65+ var userErr *domain.UserError
66+ require.ErrorAs(t, paletteErr, &userErr)
67+ require.Equal(t, domain.NoticeSessionUnavailable, userErr.Code)
68+ require.Equal(t, domain.NoticeWarn, userErr.Severity)
69+ require.Equal(t, sessionKillRetryMessage, userErr.Msg)
70+ require.ErrorIs(t, userErr.Err, errPurgeAdmissionClosed)
71+}
72+
5373 func TestMoveRejectionPresentationParity(t *testing.T) {
5474 tests := []struct {
5575 name string
M internal/usecase/daemon/move_transaction.go +7 -0
@@ -2,6 +2,7 @@ package daemon
22
33 import (
44 "context"
5+ "errors"
56
67 "github.com/bnema/vev/internal/domain"
78 )
@@ -110,6 +111,12 @@ func (d *Daemon) executeMove(topology moveTopology) (result error) {
110111
111112 reservation, err := d.reserveMoveLifecycles(t.source, t.destination)
112113 if err != nil {
114+ // Preserve a typed transient admission rejection (a KillAll purge owning
115+ // the exact set) so the command path maps it retryably instead of
116+ // misreporting the destination as gone.
117+ if errors.Is(err, errPurgeAdmissionClosed) {
118+ return err
119+ }
113120 return errMoveStaleTarget
114121 }
115122 reservationHeld := true
M internal/usecase/daemon/notify.go +21 -7
@@ -182,12 +182,20 @@ func noticeDetails(err error) string {
182182 return strings.Join(parts, " ← ")
183183 }
184184
185-// benignNoticeError reports errors that are expected control flow rather than
186-// something the user needs to be told about.
185+// benignNoticeError reports unclassified errors that are expected control flow
186+// rather than something the user needs to be told about. A caller that wraps one
187+// of these in an explicit *domain.UserError still reports its own message; this
188+// filter only keeps an unclassified occurrence from surfacing as an
189+// internal-error notice.
187190 func benignNoticeError(err error) bool {
188191 return errors.Is(err, context.Canceled) ||
189192 errors.Is(err, errNoNeighbor) ||
190- errors.Is(err, ports.ErrNoClipboardImage)
193+ errors.Is(err, ports.ErrNoClipboardImage) ||
194+ // A concurrent detach or token resume that invalidated a teardown's
195+ // participant snapshot is expected control flow: the caller observes an
196+ // explicit error and retries or reports it, so it must not also surface
197+ // as an internal-error notice.
198+ errors.Is(err, errSessionKillParticipantsChanged)
191199 }
192200
193201 // persistShutdownSnapshotFailure records a terminal snapshot failure that
@@ -222,17 +230,23 @@ func (d *Daemon) persistShutdownSnapshotFailure(name string, cause error) {
222230 }
223231 }
224232
225-// reportError turns any error into a user-facing notice. Unclassified errors
226-// become NoticeInternal: an error reaching here is never silently dropped
227-// unless benignNoticeError says it is routine.
233+// reportError turns any error into a user-facing notice. An explicitly
234+// classified *domain.UserError always wins: the caller already decided what the
235+// user should see, so it is reported even when the cause chain also matches a
236+// routine sentinel (for example a retryable teardown abort). Any other error is
237+// dropped only when benignNoticeError says it is routine; unclassified errors
238+// become NoticeInternal.
228239 func (d *Daemon) reportError(sess *session, err error) {
229- if err == nil || benignNoticeError(err) {
240+ if err == nil {
230241 return
231242 }
232243 if ue, ok := errors.AsType[*domain.UserError](err); ok {
233244 d.notify(sess, ue.Severity, ue.Code, ue.Msg, ue.Err)
234245 return
235246 }
247+ if benignNoticeError(err) {
248+ return
249+ }
236250 d.notify(sess, domain.NoticeError, domain.NoticeInternal, "internal error", err)
237251 }
238252
M internal/usecase/daemon/notify_test.go +18 -0
@@ -938,3 +938,21 @@ func TestToastExpiryRemovesOnlyTheEntryItArmed(t *testing.T) {
938938 return len(ns) == 0
939939 }, 200*time.Millisecond, 5*time.Millisecond, "expiry must remove only the entry it armed")
940940 }
941+
942+// TestReportErrorClassifiedUserErrorWinsOverBenignCause proves an explicitly
943+// classified user error is reported even when its cause chain also matches a
944+// routine sentinel. A retryable teardown abort must reach the user as its own
945+// message instead of being filtered as internal control flow.
946+func TestReportErrorClassifiedUserErrorWinsOverBenignCause(t *testing.T) {
947+ d, sess, ac, _ := newNoticeFixture(t, newNoticeClock())
948+
949+ d.reportError(sess, closeTabFailureNotice(fmt.Errorf("close tab: %w", errSessionKillParticipantsChanged)))
950+
951+ hist := d.notices.history()
952+ require.Len(t, hist, 1)
953+ require.Contains(t, hist[0].Message, "couldn't close tab")
954+ require.Contains(t, hist[0].Message, "try again")
955+
956+ toasts := awaitToastCount(t, ac, 1)
957+ require.Contains(t, toasts[0].Message, "try again")
958+}
M internal/usecase/daemon/pane_actions.go +16 -5
@@ -315,17 +315,28 @@ func (d *Daemon) reapTiledPaneLease(lease paneEffectLease) bool {
315315 ac.overlays.clearCopyModeForPane(p)
316316 }
317317 }
318+ // A concurrent detach or token resume that invalidated the teardown's
319+ // participant snapshot is itself observable progress: the next owner
320+ // resolution consumes the fresh participant set, so retry instead of
321+ // treating the abort as a completed close.
322+ participantsChanged := false
318323 if err := d.closeTabLocked(sess, tb, true); err != nil {
319- return true
324+ if !errors.Is(err, errSessionKillParticipantsChanged) {
325+ return true
326+ }
327+ participantsChanged = true
320328 }
321329 if p.owner.Load() == nil {
322330 return true
323331 }
324332 // A successful no-op teardown means participant, gate, or lifecycle
325333 // validation rejected the mutation without changing either the tab list
326- // or the pane owner. Retrying that exact state would spin forever; only
327- // retry when the teardown made observable progress that can be consumed
328- // by the next owner resolution.
334+ // or the pane owner. That includes an explicit participants-changed
335+ // sentinel: the kill aborted because a concurrent detach or transport
336+ // replacement invalidated its snapshot, which the owner resolution below
337+ // consumes. Retrying that exact state would spin forever; only retry when
338+ // the teardown made observable progress that can be consumed by the next
339+ // owner resolution.
329340 d.mu.Lock()
330341 sessionCurrent := d.sessions[sess.id] == sess
331342 sess.mu.Lock()
@@ -341,7 +352,7 @@ func (d *Daemon) reapTiledPaneLease(lease paneEffectLease) bool {
341352 }
342353 sess.mu.Unlock()
343354 d.mu.Unlock()
344- if sessionCurrent && tabCurrent && !attachmentsChanged {
355+ if sessionCurrent && tabCurrent && !attachmentsChanged && !participantsChanged {
345356 return true
346357 }
347358 return false
M internal/usecase/daemon/picker.go +6 -0
@@ -726,6 +726,9 @@ func (d *Daemon) resumeStoppedAndSwitchLocked(from *session, ac *attachedClient,
726726 expectedTransport: sourceCapability.transport, sourceCapability: &sourceCapability, sourceEffect: sourceEffect, action: action,
727727 expectedSourceTab: guard.expectedSource, copySourceEnvironment: true, ready: true,
728728 createTargetLocked: func() (*session, error) {
729+ if d.purgeAdmissionClosedLocked() {
730+ return nil, errPurgeAdmissionClosed
731+ }
729732 current, ok := d.inactive[target.Name]
730733 if !ok || !current.canResume() || !current.sameLifecycle(stopped) || !targetMatchesLifecycle(target, current.name, current.createdAt, current.incarnation) {
731734 return nil, errAttachmentTransition
@@ -759,6 +762,9 @@ func (d *Daemon) resumeStoppedAndSwitchLocked(from *session, ac *attachedClient,
759762 }
760763 env := copyEnvironment(from.env)
761764 from.mu.Unlock()
765+ if d.purgeAdmissionClosedLocked() {
766+ return nil, attachmentTransitionResult{}, false, errPurgeAdmissionClosed
767+ }
762768 current, ok := d.inactive[target.Name]
763769 if !ok || !current.canResume() || !current.sameLifecycle(stopped) || !targetMatchesLifecycle(target, current.name, current.createdAt, current.incarnation) {
764770 return nil, attachmentTransitionResult{}, false, errAttachmentTransition
A internal/usecase/daemon/purge.go +681 -0
@@ -0,0 +1,681 @@
1+package daemon
2+
3+import (
4+ "context"
5+ "errors"
6+ "fmt"
7+ "sort"
8+ "strings"
9+ "sync"
10+
11+ "github.com/bnema/vev/internal/domain"
12+)
13+
14+// errPurgeAdmissionClosed is returned to command-path creation/resume that
15+// already holds architecture locks and therefore cannot wait for a daemon-wide
16+// purge. Route-level creation waits on the purge admission gate instead.
17+var errPurgeAdmissionClosed = errors.New("daemon is purging sessions")
18+
19+// purgeAdmissionClosedLocked reports whether a KillAll purge currently owns
20+// admission. Caller holds mu. A command-path create/resume that cannot wait
21+// rejects with errPurgeAdmissionClosed instead of inserting into the exact set
22+// being removed.
23+func (d *Daemon) purgeAdmissionClosedLocked() bool { return d.purgeAdmissionClosing }
24+
25+// acquirePurgeAdmission reserves one create/restore transition against the
26+// transient KillAll gate. It waits without holding any other lock while a purge
27+// owns the gate, so a Hello arriving mid-purge is admitted as a create or
28+// restore after the purge rather than inserted into the set being removed.
29+func (d *Daemon) acquirePurgeAdmission(ctx context.Context) error {
30+ for {
31+ d.mu.Lock()
32+ if !d.purgeAdmissionClosing {
33+ d.purgeAdmissionActive++
34+ d.mu.Unlock()
35+ return nil
36+ }
37+ changed := d.purgeAdmissionChangeLocked()
38+ d.mu.Unlock()
39+ select {
40+ case <-changed:
41+ case <-ctx.Done():
42+ return ctx.Err()
43+ }
44+ }
45+}
46+
47+// releasePurgeAdmission drops a reservation taken by acquirePurgeAdmission or
48+// an equivalent move reservation.
49+func (d *Daemon) releasePurgeAdmission() {
50+ d.mu.Lock()
51+ d.purgeAdmissionActive--
52+ d.signalPurgeAdmissionChangedLocked()
53+ d.mu.Unlock()
54+}
55+
56+// purgeAdmissionStatus reports how a KillAll's attempt to close the transient
57+// admission gate resolved.
58+type purgeAdmissionStatus uint8
59+
60+const (
61+ // purgeAdmissionGranted means the gate closed and every admitted transition
62+ // drained within the bound, so the purge owns the exact lifecycle set.
63+ purgeAdmissionGranted purgeAdmissionStatus = iota
64+ // purgeAdmissionSuperseded means an explicit daemon shutdown began, or the
65+ // caller's context ended, before the drain completed. The purge removes
66+ // nothing and defers to the stop.
67+ purgeAdmissionSuperseded
68+ // purgeAdmissionTimedOut means an admitted transition did not release the
69+ // gate before the bounded purge budget expired. The purge removes nothing so
70+ // a transition stuck in repository I/O that ignores cancellation can never
71+ // wedge Serve.
72+ purgeAdmissionTimedOut
73+)
74+
75+// beginPurgeAdmission closes the KillAll admission gate and waits, bounded, for
76+// every admitted transition to drain before the caller snapshots the exact
77+// lifecycle set. It mirrors closeMoveLifecycles but is re-openable: the caller
78+// must always call endPurgeAdmission to clear it and leave the daemon serving.
79+//
80+// The wait ends as soon as the drain completes. It also ends early, without the
81+// purge owning anything, when ctx is done, an explicit shutdown begins
82+// (closing), or deadline expires — so an admitted create, restore, or move that
83+// ignores cancellation (for example a repository call that never returns) can
84+// never extend Serve past the bounded purge budget. On success purgeEpoch is
85+// incremented so a restoration worker that waited on the gate can detect that
86+// the record set it captured was superseded.
87+func (d *Daemon) beginPurgeAdmission(ctx context.Context, deadline *snapshotShutdownDeadline) purgeAdmissionStatus {
88+ if ctx == nil {
89+ ctx = context.Background()
90+ }
91+ d.mu.Lock()
92+ d.purgeAdmissionClosing = true
93+ d.signalPurgeAdmissionChangedLocked()
94+ for {
95+ if d.closing {
96+ d.mu.Unlock()
97+ return purgeAdmissionSuperseded
98+ }
99+ if d.purgeAdmissionActive == 0 {
100+ d.purgeEpoch++
101+ d.signalPurgeAdmissionChangedLocked()
102+ d.mu.Unlock()
103+ return purgeAdmissionGranted
104+ }
105+ changed := d.purgeAdmissionChangeLocked()
106+ d.mu.Unlock()
107+ select {
108+ case <-changed:
109+ case <-ctx.Done():
110+ return purgeAdmissionSuperseded
111+ case <-deadline.Done():
112+ // An explicit stop that began as the budget expired wins: classify it
113+ // as superseded so the purge defers and reports stop semantics instead
114+ // of a drain timeout. terminateAllForShutdown also signals changed, but
115+ // a ready select may still pick the deadline, so recheck closing here.
116+ d.mu.Lock()
117+ closing := d.closing
118+ d.mu.Unlock()
119+ if closing {
120+ return purgeAdmissionSuperseded
121+ }
122+ return purgeAdmissionTimedOut
123+ }
124+ d.mu.Lock()
125+ }
126+}
127+
128+func (d *Daemon) endPurgeAdmission() {
129+ d.mu.Lock()
130+ d.purgeAdmissionClosing = false
131+ d.signalPurgeAdmissionChangedLocked()
132+ d.mu.Unlock()
133+}
134+
135+// purgeEpochSnapshot returns the current purge boundary. Restoration captures
136+// it before reading catalogue records so a purge that completes while its
137+// workers wait for admission invalidates those records.
138+func (d *Daemon) purgeEpochSnapshot() uint64 {
139+ d.mu.Lock()
140+ defer d.mu.Unlock()
141+ return d.purgeEpoch
142+}
143+
144+// purgeSupersededLocked reports whether an admitted transition captured at
145+// epoch must abandon its work: an explicit shutdown has begun, a purge currently
146+// owns and waits on admission, or a purge completed after epoch was captured.
147+// Caller holds mu.
148+func (d *Daemon) purgeSupersededLocked(epoch uint64) bool {
149+ return d.closing || d.purgeAdmissionClosing || d.purgeEpoch != epoch
150+}
151+
152+// purgeSuperseded is purgeSupersededLocked without the caller-held lock.
153+func (d *Daemon) purgeSuperseded(epoch uint64) bool {
154+ d.mu.Lock()
155+ defer d.mu.Unlock()
156+ return d.purgeSupersededLocked(epoch)
157+}
158+
159+// purgeSupersededForShutdown reports whether an explicit daemon shutdown has
160+// begun. A KillAll purge defers to it so explicit stop semantics (preserving
161+// live sessions as stopped authority) are never overwritten by a concurrent
162+// purge.
163+func (d *Daemon) purgeSupersededForShutdown() bool {
164+ d.mu.Lock()
165+ defer d.mu.Unlock()
166+ return d.closing
167+}
168+
169+// purgeAdmissionChangeLocked returns the channel closed when either purge
170+// admission field changes. Caller holds mu.
171+func (d *Daemon) purgeAdmissionChangeLocked() chan struct{} {
172+ if d.purgeAdmissionChanged == nil {
173+ d.purgeAdmissionChanged = make(chan struct{})
174+ }
175+ return d.purgeAdmissionChanged
176+}
177+
178+// signalPurgeAdmissionChangedLocked wakes every admission waiter. Caller holds
179+// mu.
180+func (d *Daemon) signalPurgeAdmissionChangedLocked() {
181+ close(d.purgeAdmissionChangeLocked())
182+ d.purgeAdmissionChanged = make(chan struct{})
183+}
184+
185+// purgeRecordClass identifies which lifecycle record class a failed purge unit
186+// belongs to, so callers can report a typed partial failure instead of one
187+// opaque error.
188+type purgeRecordClass uint8
189+
190+const (
191+ purgeRecordLive purgeRecordClass = iota + 1
192+ purgeRecordStopped
193+ purgeRecordBroken
194+ purgeRecordDurable
195+)
196+
197+func (c purgeRecordClass) String() string {
198+ switch c {
199+ case purgeRecordLive:
200+ return "live"
201+ case purgeRecordStopped:
202+ return "stopped"
203+ case purgeRecordBroken:
204+ return "broken"
205+ case purgeRecordDurable:
206+ return "durable"
207+ default:
208+ return "unknown"
209+ }
210+}
211+
212+// inactivePurgeClass resolves the record class of a captured stopped/broken
213+// registry entry.
214+func inactivePurgeClass(entry inactiveSession) purgeRecordClass {
215+ if entry.broken() {
216+ return purgeRecordBroken
217+ }
218+ return purgeRecordStopped
219+}
220+
221+// purgeFailure is one structured partial purge failure: the record class, the
222+// affected name when known, and the underlying error.
223+type purgeFailure struct {
224+ Class purgeRecordClass
225+ Name string
226+ Err error
227+}
228+
229+func (f purgeFailure) Error() string {
230+ if f.Name == "" {
231+ return fmt.Sprintf("%s: %v", f.Class, f.Err)
232+ }
233+ return fmt.Sprintf("%s %q: %v", f.Class, f.Name, f.Err)
234+}
235+
236+func (f purgeFailure) Unwrap() error { return f.Err }
237+
238+// purgeResult reports every record that could not be fully purged. An empty
239+// Failures slice means the purge removed the entire exact lifecycle set. A
240+// superseded result means an explicit daemon shutdown won the race: the purge
241+// deferred and removed nothing, or removed only the units whose teardown it had
242+// already owned when the stop began. A drain timeout result means an admitted
243+// transition refused to drain within the bounded admission budget, so the purge
244+// removed nothing and reopened admission. A typed live failure means a live unit
245+// was left untouched because the shared live-teardown budget expired before its
246+// destructive boundary or because its participants kept changing. A typed
247+// stopped/broken failure means the bounded stopped/broken sweep did not finish
248+// the record's durable purge within its own budget.
249+type purgeResult struct {
250+ Failures []purgeFailure
251+ superseded bool
252+ drainTimedOut bool
253+}
254+
255+func (r purgeResult) Failed() bool { return len(r.Failures) > 0 || r.drainTimedOut }
256+
257+// Superseded reports that an explicit daemon shutdown owned the lifecycle set
258+// before this purge could finish destroying it, so the purge deferred to it.
259+// Any Failures it carries describe units the stop still owns; a superseded
260+// result never means the purge silently reported success.
261+func (r purgeResult) Superseded() bool { return r.superseded }
262+
263+// purgeSummaryFailureLimit bounds the wire-safe failure detail. The result may
264+// hold arbitrarily many failures (one per record), but a control response must
265+// stay bounded: report the total count and only the first N failures, then note
266+// how many were omitted.
267+const purgeSummaryFailureLimit = 5
268+
269+func (r purgeResult) summary() string {
270+ if r.drainTimedOut {
271+ return "session purge aborted: purge admission did not drain within the bounded timeout; no records were removed"
272+ }
273+ if !r.Failed() {
274+ return ""
275+ }
276+ shown := min(len(r.Failures), purgeSummaryFailureLimit)
277+ parts := make([]string, 0, shown)
278+ for _, failure := range r.Failures[:shown] {
279+ parts = append(parts, failure.Error())
280+ }
281+ if omitted := len(r.Failures) - shown; omitted > 0 {
282+ return fmt.Sprintf("%d record(s) could not be purged; first %d: %s; %d more omitted", len(r.Failures), shown, strings.Join(parts, "; "), omitted)
283+ }
284+ return fmt.Sprintf("%d record(s) could not be purged: %s", len(r.Failures), strings.Join(parts, "; "))
285+}
286+
287+// purgeAllSessions is the KillAll operation: it removes every live session and
288+// every stopped/broken/durable record while leaving the daemon active. It does
289+// not set closing, close move admission, cancel daemon-wide pane/process
290+// contexts, close done, or consume shutdown signals, and it never makes Serve
291+// return.
292+//
293+// Linearization: beginPurgeAdmission closes the transient admission gate and
294+// drains in-flight create/restore/move reservations, so the exact lifecycle set
295+// captured below cannot be mutated by a concurrent transition. Purge then runs
296+// against that exact set, and endPurgeAdmission reopens admission for later
297+// creations, which are ordered after the purge.
298+//
299+// Budgets: the admission drain, the stopped/broken durable sweep, and
300+// live-session teardown spend separate bounded budgets, each matching the
301+// daemon shutdown budget. A transition that ignores
302+// cancellation and burns the whole drain budget therefore cannot leave teardown
303+// with no time, and teardown cannot extend the drain. A repository deletion
304+// that ignores cancellation cannot extend the stopped/broken sweep past its own
305+// budget either. The live budget is armed
306+// when the live phase begins and is shared by every live unit: once it expires,
307+// each remaining live unit is skipped and reported as a typed live failure
308+// rather than reported as purged. Every unit keeps its own accurate name.
309+func (d *Daemon) purgeAllSessions(reason uint8) purgeResult {
310+ d.purgeAllMu.Lock()
311+ defer d.purgeAllMu.Unlock()
312+
313+ // One deadline bounds the drain of every admitted transition. It is stopped
314+ // before the destructive phase so teardown spends a fresh budget.
315+ admissionDeadline := newSnapshotShutdownDeadline(d.clock)
316+ defer admissionDeadline.stop()
317+
318+ ctx := d.serveCtx
319+ if ctx == nil {
320+ ctx = context.Background()
321+ }
322+ switch d.beginPurgeAdmission(ctx, admissionDeadline) {
323+ case purgeAdmissionSuperseded:
324+ d.log.Info("session purge superseded before admission", "reason", reason)
325+ d.endPurgeAdmission()
326+ return purgeResult{superseded: true}
327+ case purgeAdmissionTimedOut:
328+ d.log.Warn("session purge aborted: purge admission did not drain", "reason", reason)
329+ d.endPurgeAdmission()
330+ return purgeResult{drainTimedOut: true}
331+ }
332+ admissionDeadline.stop()
333+ defer d.endPurgeAdmission()
334+
335+ d.mu.Lock()
336+ // A shutdown that began after admission was granted still wins: no record
337+ // has been removed yet, so defer without touching the preserved set.
338+ if d.closing {
339+ d.mu.Unlock()
340+ d.log.Info("session purge superseded by daemon shutdown", "reason", reason, "phase", "pre-snapshot")
341+ return purgeResult{superseded: true}
342+ }
343+ d.purgeAllParkingLocked()
344+ parkedRetirements := d.purgeAllParkedLocked()
345+ d.purgeAllSuspendedLocked()
346+ live := d.sessionsSnapshotLocked()
347+ inactive := make([]inactiveSession, 0, len(d.inactive))
348+ exact := make(map[string]struct{}, len(live)+len(d.inactive))
349+ for _, entry := range d.inactive {
350+ inactive = append(inactive, entry)
351+ exact[entry.name] = struct{}{}
352+ }
353+ for _, sess := range live {
354+ sess.mu.Lock()
355+ exact[sess.name] = struct{}{}
356+ sess.mu.Unlock()
357+ }
358+ d.mu.Unlock()
359+ if d.afterPurgeAdmission != nil {
360+ d.afterPurgeAdmission()
361+ }
362+ d.finishParkedAttachmentRetirements(parkedRetirements)
363+ sort.Slice(inactive, func(i, j int) bool { return inactive[i].name < inactive[j].name })
364+
365+ // A shutdown that began while the snapshot was taken still wins: no record
366+ // has been removed yet, so defer without touching the preserved set.
367+ if d.purgeSupersededForShutdown() {
368+ d.log.Info("session purge superseded by daemon shutdown", "reason", reason, "phase", "pre-destructive")
369+ return purgeResult{superseded: true}
370+ }
371+
372+ result := purgeResult{}
373+ d.log.Info("session purge begin", "reason", reason, "live_sessions", len(live), "inactive_sessions", len(inactive))
374+
375+ // Stopped authority first (mirroring the shutdown purge order), then live
376+ // sessions. Each unit is independent: one failure never cancels a sibling.
377+ // Explicit stop wins per unit: a shutdown that begins mid-purge stops further
378+ // removals so it can preserve every not-yet-purged lifecycle as stopped
379+ // authority. A unit the purge already owns (its teardown is in progress)
380+ // keeps purge authority and completes as a purge; only units not yet started
381+ // are handed back to the stop.
382+ //
383+ // The stopped/broken sweep spends its own deadline and context, separate from
384+ // the admission drain and the live-teardown budget. A repository deletion
385+ // that ignores cancellation therefore cannot extend the purge: at the budget
386+ // every record the sweep did not finish is still stopped/broken authority and
387+ // is reported as a typed failure, and admission is released promptly.
388+ if len(inactive) > 0 {
389+ stoppedDeadline := newSnapshotShutdownDeadline(d.clock)
390+ defer stoppedDeadline.stop()
391+ stoppedCtx, cancelStopped := snapshotStopContext(stoppedDeadline)
392+ defer cancelStopped()
393+ sweep := d.purgeInactiveSweep(inactive, stoppedCtx, stoppedDeadline)
394+ result.Failures = append(result.Failures, sweep.failures...)
395+ if sweep.superseded {
396+ d.log.Info("session purge superseded by daemon shutdown", "reason", reason, "phase", "stopped")
397+ return purgeResult{Failures: result.Failures, superseded: true}
398+ }
399+ }
400+ // The live phase gets its own bounded budget, armed here so neither the
401+ // admission drain nor the stopped-record purge consumes it. It is shared by
402+ // every live unit.
403+ destructiveDeadline := newSnapshotShutdownDeadline(d.clock)
404+ defer destructiveDeadline.stop()
405+ for _, sess := range live {
406+ if d.purgeSupersededForShutdown() {
407+ d.log.Info("session purge superseded by daemon shutdown", "reason", reason, "phase", "live")
408+ return purgeResult{Failures: result.Failures, superseded: true}
409+ }
410+ sess.mu.Lock()
411+ name := sess.name
412+ sess.mu.Unlock()
413+ if d.beforePurgeLiveSessionKill != nil {
414+ d.beforePurgeLiveSessionKill(sess)
415+ }
416+ sess.stopInMemoryLifecycle()
417+ err := d.purgeLiveSession(sess, reason, destructiveDeadline)
418+ if err == nil {
419+ // A stop that wins this unit after the precheck above makes
420+ // purgeLiveSession a no-op: the stop already removed the session and
421+ // now owns it. Recheck so the purge never counts a stop-owned unit as a
422+ // successful purge. A unit whose teardown the purge already owned keeps
423+ // purge authority and is reported as removed, but the remaining units
424+ // still belong to the stop.
425+ if d.purgeSupersededForShutdown() {
426+ d.log.Info("session purge superseded by daemon shutdown", "reason", reason, "phase", "live-owned")
427+ return purgeResult{Failures: result.Failures, superseded: true}
428+ }
429+ continue
430+ }
431+ if d.purgeSupersededForShutdown() &&
432+ (errors.Is(err, errSessionKillDeadline) || errors.Is(err, errSessionKillParticipantsChanged)) {
433+ // The explicit stop owns the remaining lifecycles: either the live budget
434+ // expired as it began, or a participant change aborted a kill that the
435+ // stop now classifies as preservation rather than a purge failure.
436+ d.log.Info("session purge superseded by daemon shutdown", "reason", reason, "phase", "live-deadline")
437+ return purgeResult{Failures: result.Failures, superseded: true}
438+ }
439+ result.Failures = append(result.Failures, purgeFailure{Class: purgeRecordLive, Name: name, Err: err})
440+ d.log.Error("purging live session failed", "err", err, "session", name)
441+ }
442+ result.Failures = append(result.Failures, d.purgeDurableResidue(exact, result.Failures)...)
443+ sort.SliceStable(result.Failures, func(i, j int) bool {
444+ if result.Failures[i].Class != result.Failures[j].Class {
445+ return result.Failures[i].Class < result.Failures[j].Class
446+ }
447+ return result.Failures[i].Name < result.Failures[j].Name
448+ })
449+ if result.Failed() {
450+ d.log.Warn("session purge incomplete", "reason", reason, "failures", len(result.Failures))
451+ }
452+ return result
453+}
454+
455+// purgeDurableResidue removes durable records that have no live/stopped/broken
456+// registry entry, so a KillAll still accounts for every catalogue record. A
457+// record already represented by a reported live/stopped/broken failure is that
458+// failure's durable residue and is not double-reported here.
459+func (d *Daemon) purgeDurableResidue(exact map[string]struct{}, existing []purgeFailure) []purgeFailure {
460+ if !d.persistEnabled || d.catalogue == nil {
461+ return nil
462+ }
463+ records, err := d.catalogue.Records()
464+ if err != nil {
465+ return []purgeFailure{{Class: purgeRecordDurable, Err: err}}
466+ }
467+ reported := make(map[string]struct{}, len(existing))
468+ for _, failure := range existing {
469+ reported[failure.Name] = struct{}{}
470+ }
471+ var out []purgeFailure
472+ for _, record := range records {
473+ if _, accounted := exact[record.Name]; accounted {
474+ continue
475+ }
476+ if _, already := reported[record.Name]; already {
477+ continue
478+ }
479+ if err := d.purgeDurableRecord(record); err != nil {
480+ out = append(out, purgeFailure{Class: purgeRecordDurable, Name: record.Name, Err: err})
481+ }
482+ }
483+ return out
484+}
485+
486+// errSessionPurgeSweepDeadline reports that the stopped/broken sweep's own
487+// bounded budget expired before a record's durable purge finished. The record
488+// keeps its stopped/broken authority and is reported as a typed failure rather
489+// than silently counted as purged.
490+var errSessionPurgeSweepDeadline = errors.New("stopped session purge sweep deadline exceeded")
491+
492+// errSessionPurgeDeletePending reports that a purge-owned live unit's exact
493+// durable delete did not finish within the shared live-teardown budget. The
494+// live identity is already removed and its stopped authority still reserves the
495+// name; a detached worker finishes the fenced DeleteExact and removes that
496+// authority once the repository returns. It is a typed failure so a bounded
497+// purge never reports a skipped durable delete as success, and a repository
498+// delete that ignores cancellation can never hold purgeAllMu, the admission
499+// gate, or control responses.
500+var errSessionPurgeDeletePending = errors.New("session durable purge pending")
501+
502+// finishLiveDurablePurge runs the exact durable delete for a live unit the purge
503+// already removed from the live registry. With a bounded deadline it runs the
504+// delete on a detached worker: the caller returns errSessionPurgeDeletePending
505+// at the deadline while the worker finishes and removes the matching stopped
506+// authority. That removal is fenced by the captured name, incarnation, and
507+// createdAt, so a same-name or re-incarnated replacement is never touched. A
508+// nil deadline (a command-path kill) keeps the previous synchronous behavior.
509+func (d *Daemon) finishLiveDurablePurge(deadline *snapshotShutdownDeadline, record domain.CatalogueRecord) error {
510+ expected := inactiveSession{name: record.Name, createdAt: record.CreatedAt, incarnation: record.IncarnationID, purging: true}
511+ finish := func() error {
512+ if err := d.finishSnapshotPurge(d.serveCtx, record.Name, record.IncarnationID, record.CreatedAt); err != nil {
513+ return err
514+ }
515+ d.mu.Lock()
516+ if stopped, ok := d.inactive[record.Name]; ok && stopped.purging && stopped.sameLifecycle(expected) {
517+ delete(d.inactive, record.Name)
518+ }
519+ d.mu.Unlock()
520+ d.reconcileAllRouteHistories()
521+ return nil
522+ }
523+ if deadline == nil {
524+ return finish()
525+ }
526+ done := make(chan error, 1)
527+ go func() { done <- finish() }()
528+ select {
529+ case err := <-done:
530+ return err
531+ case <-deadline.Done():
532+ // Recheck completion so a delete that finished as the budget expired is
533+ // reported as success rather than pending.
534+ select {
535+ case err := <-done:
536+ return err
537+ default:
538+ // The caller already reported a typed pending failure and stopped
539+ // waiting. Wait for the detached delete on its own goroutine so a late
540+ // failure is still recorded with bounded, non-sensitive fields (record
541+ // class, incarnation, created-time) instead of the user-chosen session
542+ // name.
543+ go func() {
544+ if err := <-done; err != nil {
545+ d.log.Warn("detached session purge delete failed after deadline",
546+ "class", purgeRecordLive, "incarnation", record.IncarnationID, "created_at", record.CreatedAt, "err", err)
547+ }
548+ }()
549+ return errSessionPurgeDeletePending
550+ }
551+ }
552+}
553+
554+// purgeInactiveSweepResult reports the bounded stopped/broken sweep outcome.
555+// failures holds every record the purge could not finish; superseded means an
556+// explicit stop began mid-sweep and owns the remaining records.
557+type purgeInactiveSweepResult struct {
558+ failures []purgeFailure
559+ superseded bool
560+}
561+
562+// purgeInactiveSweep removes the captured stopped/broken records under its own
563+// deadline and context, separate from the admission drain and the
564+// live-teardown budget. Because a repository deletion may ignore cancellation,
565+// the sweep runs on a detached worker and the caller returns at the deadline:
566+// every record the worker did not finish keeps its stopped/broken authority and
567+// is reported as a typed failure, so a bounded purge releases admission
568+// promptly instead of waiting out an uncooperative delete. The detached worker
569+// stops contributing to the returned result at that point; its registry removal
570+// stays lifecycle-fenced, so a late completion cannot delete a same-name
571+// replacement.
572+func (d *Daemon) purgeInactiveSweep(inactive []inactiveSession, ctx context.Context, deadline *snapshotShutdownDeadline) purgeInactiveSweepResult {
573+ var (
574+ mu sync.Mutex
575+ failures []purgeFailure
576+ completed int
577+ superseded bool
578+ detached bool
579+ )
580+ sweepDone := make(chan struct{})
581+ go func() {
582+ defer close(sweepDone)
583+ for _, entry := range inactive {
584+ mu.Lock()
585+ abandoned := detached
586+ mu.Unlock()
587+ if abandoned {
588+ return
589+ }
590+ if d.purgeSupersededForShutdown() {
591+ mu.Lock()
592+ superseded = true
593+ mu.Unlock()
594+ return
595+ }
596+ class := inactivePurgeClass(entry)
597+ err := d.retryStoppedPurgeContextExact(ctx, entry.name, entry.incarnation, &entry.createdAt)
598+ mu.Lock()
599+ if detached {
600+ // The bounded caller already reported this unit; a late result must
601+ // not mutate the returned result.
602+ mu.Unlock()
603+ return
604+ }
605+ if err != nil {
606+ failures = append(failures, purgeFailure{Class: class, Name: entry.name, Err: err})
607+ d.log.Error("purging stopped session failed", "err", err, "session", entry.name)
608+ }
609+ completed++
610+ mu.Unlock()
611+ }
612+ }()
613+
614+ select {
615+ case <-sweepDone:
616+ case <-deadline.Done():
617+ mu.Lock()
618+ detached = true
619+ done := completed
620+ reported := append([]purgeFailure(nil), failures...)
621+ mu.Unlock()
622+ if d.purgeSupersededForShutdown() {
623+ // An explicit stop that began as the budget expired owns the remaining
624+ // records, so the sweep is supersession rather than a purge failure.
625+ return purgeInactiveSweepResult{failures: reported, superseded: true}
626+ }
627+ for _, entry := range inactive[done:] {
628+ reported = append(reported, purgeFailure{Class: inactivePurgeClass(entry), Name: entry.name, Err: errSessionPurgeSweepDeadline})
629+ d.log.Error("purging stopped session failed", "err", errSessionPurgeSweepDeadline, "session", entry.name)
630+ }
631+ return purgeInactiveSweepResult{failures: reported}
632+ }
633+
634+ mu.Lock()
635+ defer mu.Unlock()
636+ return purgeInactiveSweepResult{failures: failures, superseded: superseded}
637+}
638+
639+// purgeLiveSession removes one live session under the shared live-teardown
640+// budget. A post-freeze participant change aborts the kill before it touches the
641+// session. A session a racing path already removed owns nothing and counts as
642+// purged; an explicit stop that began meanwhile owns the rest and is never
643+// retried against; otherwise the purge retries once against a fresh snapshot,
644+// and a session whose participants changed again stays a typed live failure for
645+// the caller.
646+func (d *Daemon) purgeLiveSession(sess *session, reason uint8, deadline *snapshotShutdownDeadline) error {
647+ err := d.killSessionWithSnapshotDeadline(sess, reason, true, deadline, nil)
648+ if !errors.Is(err, errSessionKillParticipantsChanged) {
649+ return err
650+ }
651+ if d.purgeSupersededForShutdown() {
652+ // An explicit stop has begun: it owns the session's remaining lifecycle, so
653+ // the purge must not retry against it. The caller classifies the abort as
654+ // supersession instead of a live failure.
655+ return err
656+ }
657+ if !d.sessionRegistered(sess) {
658+ return nil
659+ }
660+ return d.killSessionWithSnapshotDeadline(sess, reason, true, deadline, nil)
661+}
662+
663+// sessionRegistered reports whether sess is still the exact live registry entry.
664+func (d *Daemon) sessionRegistered(sess *session) bool {
665+ if sess == nil {
666+ return false
667+ }
668+ d.mu.Lock()
669+ defer d.mu.Unlock()
670+ return d.sessions[sess.id] == sess
671+}
672+
673+func (d *Daemon) purgeDurableRecord(record domain.CatalogueRecord) error {
674+ if !d.persistEnabled {
675+ return nil
676+ }
677+ if d.recovery == nil {
678+ return errors.New("durable session authority is not configured")
679+ }
680+ return d.finishSnapshotPurge(d.serveCtx, record.Name, record.IncarnationID, record.CreatedAt)
681+}
A internal/usecase/daemon/purge_test.go +2386 -0
@@ -0,0 +1,2386 @@
1+package daemon
2+
3+import (
4+ "bytes"
5+ "context"
6+ "errors"
7+ "fmt"
8+ "log/slog"
9+ "strings"
10+ "sync"
11+ "sync/atomic"
12+ "testing"
13+ "time"
14+
15+ "github.com/stretchr/testify/mock"
16+ "github.com/stretchr/testify/require"
17+
18+ "github.com/bnema/vev/internal/domain"
19+ "github.com/bnema/vev/internal/ports"
20+ portsmocks "github.com/bnema/vev/internal/ports/mocks"
21+ "github.com/bnema/vev/internal/protocol"
22+ recoveryusecase "github.com/bnema/vev/internal/usecase/recovery"
23+ snapcodec "github.com/bnema/vev/internal/usecase/snapshot"
24+)
25+
26+// selectivePurgeRepository fails incarnation deletion only for the configured
27+// lifecycles, so a purge can succeed for some records and fail for others. It
28+// can also run a one-shot onDelete hook to interleave a state change (such as
29+// an explicit shutdown) after the first destructive purge unit.
30+type selectivePurgeRepository struct {
31+ noOpSnapshotRepository
32+
33+ mu sync.Mutex
34+ fail map[domain.IncarnationID]error
35+ calls []domain.IncarnationID
36+ onDelete func()
37+ once sync.Once
38+}
39+
40+func (r *selectivePurgeRepository) DeleteIncarnation(_ context.Context, id domain.IncarnationID) error {
41+ r.mu.Lock()
42+ r.calls = append(r.calls, id)
43+ err := r.fail[id]
44+ hook := r.onDelete
45+ r.mu.Unlock()
46+ if hook != nil {
47+ r.once.Do(hook)
48+ }
49+ return err
50+}
51+
52+func newSelectivePurgeRepository(fail map[domain.IncarnationID]error) *selectivePurgeRepository {
53+ return &selectivePurgeRepository{fail: fail}
54+}
55+
56+// TestPurgeAllRemovesLiveAndInactiveAndKeepsDaemonActive is the KillAll
57+// survival contract: every live, stopped, and broken record is removed, the
58+// daemon-wide shutdown facts are untouched, and the same daemon creates and
59+// drives new named and ephemeral sessions.
60+func TestPurgeAllRemovesLiveAndInactiveAndKeepsDaemonActive(t *testing.T) {
61+ sz := domain.Size{Cols: 80, Rows: 24}
62+ seedPTY, releaseSeed := newBlockingPTY(t)
63+ writes := make(chan []byte, 8)
64+ afterPTY, releaseAfter := newBlockingPTYWithWrites(t, writes)
65+ tabPTY, releaseTab := newBlockingPTY(t)
66+ d := newTestDaemon(t, newFactorySeq(t, seedPTY, afterPTY, tabPTY), stubClock{})
67+
68+ seed, err := createSessionForTest(d, "0", true, "/tmp", sz, terminalEnv{}, nil)
69+ require.NoError(t, err)
70+ require.NotNil(t, seed)
71+ d.mu.Lock()
72+ d.inactive["stopped"] = inactiveSession{name: "stopped", state: protocol.SessionDown, incarnation: domain.IncarnationID{7}}
73+ d.inactive["broken"] = inactiveSession{name: "broken", state: protocol.SessionBroken, incarnation: domain.IncarnationID{8}}
74+ d.mu.Unlock()
75+
76+ result := d.purgeAllSessions(protocol.ReasonSessionKilled)
77+ require.False(t, result.Failed(), "purge failures: %v", result.summary())
78+ require.Equal(t, 0, sessionCount(d))
79+ requirePurgeAdmissionBalanced(t, d)
80+ d.mu.Lock()
81+ require.Empty(t, d.inactive, "purge must remove stopped and broken records")
82+ closing := d.closing
83+ d.mu.Unlock()
84+ require.False(t, closing, "KillAll must not mark the daemon closing")
85+ d.moveLifecycleMu.Lock()
86+ moveClosing := d.moveLifecycleClosing
87+ d.moveLifecycleMu.Unlock()
88+ require.False(t, moveClosing, "KillAll must not close move admission")
89+ select {
90+ case <-d.done:
91+ t.Fatal("KillAll must not close done")
92+ default:
93+ }
94+ select {
95+ case <-d.paneProcessCtx.Done():
96+ t.Fatal("KillAll must not cancel the daemon-wide pane process context")
97+ default:
98+ }
99+
100+ // The same daemon admits a new named session, forwards PTY input, and opens
101+ // a tab through the ordinary key-driven command API.
102+ tr, sends, releaseConn := newConn(t,
103+ mustHello(protocol.IntentNew, "after", sz),
104+ frameInput([]byte("ping\n")),
105+ frameInput([]byte("\x1b ")),
106+ frameInput([]byte("CNT\r")),
107+ )
108+ var hg sync.WaitGroup
109+ hg.Go(func() { d.handleConn(tr) })
110+ awaitFrame(t, sends, "Welcome")
111+ awaitFrame(t, sends, "Output")
112+
113+ require.Eventually(t, func() bool {
114+ select {
115+ case written := <-writes:
116+ return bytes.Contains(written, []byte("ping"))
117+ default:
118+ return false
119+ }
120+ }, 2*time.Second, time.Millisecond, "PTY input was not forwarded after the purge")
121+ require.Eventually(t, func() bool {
122+ sessions := listSessions(t, d)
123+ return len(sessions.Sessions) == 1 && sessions.Sessions[0].Tabs == 2
124+ }, 2*time.Second, time.Millisecond, "tab was not created after the purge")
125+
126+ releaseConn()
127+ releaseSeed()
128+ releaseAfter()
129+ releaseTab()
130+ hg.Wait()
131+ d.sessWg.Wait()
132+ d.waitNotifies()
133+}
134+
135+// TestPurgeAllReportsStructuredPartialFailures proves failed stopped and
136+// broken records are reported with their class and name without cancelling the
137+// purge of the healthy live sibling, and without stopping the daemon.
138+func TestPurgeAllReportsStructuredPartialFailures(t *testing.T) {
139+ repository := newSelectivePurgeRepository(map[domain.IncarnationID]error{
140+ {2}: errors.New("stopped delete failed"),
141+ {3}: errors.New("broken delete failed"),
142+ })
143+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), stubClock{})
144+ WithSnapshotRepository(repository)(d)
145+ store, _ := newMockStore(t)
146+ WithStore(t, store)(d)
147+
148+ live := newSnapshotTestSession(t, "live", false, "/live")
149+ liveRecord := live.persistRecordLocked(1)
150+ require.NoError(t, d.catalogue.Create(liveRecord))
151+ livePTY, ok := live.tabs[0].panes["pane-1"].pty.(*portsmocks.MockPTY)
152+ require.True(t, ok)
153+ livePTY.EXPECT().Close().Return(nil).Once()
154+ d.sessions = map[domain.SessionID]*session{live.id: live}
155+
156+ stoppedRecord := domain.CatalogueRecord{Name: "stopped", IncarnationID: domain.IncarnationID{2}, CreatedAt: 7}
157+ require.NoError(t, d.catalogue.Create(stoppedRecord))
158+ d.inactive["stopped"] = inactiveSessionFromRecord(stoppedRecord, protocol.SessionDown, nil)
159+ brokenRecord := domain.CatalogueRecord{Name: "broken", IncarnationID: domain.IncarnationID{3}, CreatedAt: 8}
160+ require.NoError(t, d.catalogue.Create(brokenRecord))
161+ // A broken registry state with a still-present durable record: the purge
162+ // must classify the failure as broken, not stopped.
163+ d.inactive["broken"] = inactiveSessionFromRecord(brokenRecord, protocol.SessionBroken, nil)
164+
165+ result := d.purgeAllSessions(protocol.ReasonSessionKilled)
166+ require.True(t, result.Failed())
167+ require.Len(t, result.Failures, 2)
168+ require.Equal(t, purgeRecordStopped, result.Failures[0].Class)
169+ require.Equal(t, "stopped", result.Failures[0].Name)
170+ require.Equal(t, purgeRecordBroken, result.Failures[1].Class)
171+ require.Equal(t, "broken", result.Failures[1].Name)
172+ requirePurgeAdmissionBalanced(t, d)
173+
174+ require.Equal(t, 0, sessionCount(d), "the healthy live session must still be purged")
175+ _, ok, err := d.catalogue.Record("live")
176+ require.NoError(t, err)
177+ require.False(t, ok)
178+ d.mu.Lock()
179+ _, stoppedRemains := d.inactive["stopped"]
180+ _, brokenRemains := d.inactive["broken"]
181+ closing := d.closing
182+ d.mu.Unlock()
183+ require.True(t, stoppedRemains, "a failed durable purge must leave its hidden record for retry")
184+ require.True(t, brokenRemains, "a failed broken purge must leave its hidden record for retry")
185+ require.False(t, closing, "a partial purge must not stop the daemon")
186+}
187+
188+// TestPurgeAllReportsDurableOrphanFailure proves a durable catalogue record
189+// with no live/stopped/broken registry entry is purged and, when its deletion
190+// fails, reported as a durable-class failure.
191+func TestPurgeAllReportsDurableOrphanFailure(t *testing.T) {
192+ repository := newSelectivePurgeRepository(map[domain.IncarnationID]error{{9}: errors.New("orphan delete failed")})
193+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), stubClock{})
194+ WithSnapshotRepository(repository)(d)
195+ store, _ := newMockStore(t)
196+ WithStore(t, store)(d)
197+
198+ orphan := domain.CatalogueRecord{Name: "orphan", IncarnationID: domain.IncarnationID{9}, CreatedAt: 3}
199+ require.NoError(t, d.catalogue.Create(orphan))
200+
201+ result := d.purgeAllSessions(protocol.ReasonSessionKilled)
202+ require.True(t, result.Failed())
203+ require.Len(t, result.Failures, 1)
204+ require.Equal(t, purgeRecordDurable, result.Failures[0].Class)
205+ require.Equal(t, "orphan", result.Failures[0].Name)
206+}
207+
208+// TestPurgeAdmissionGatesAndReopens proves the transient gate: while a purge
209+// holds admission a route waits and a move is rejected, and after the gate is
210+// reopened both proceed.
211+func TestPurgeAdmissionGatesAndReopens(t *testing.T) {
212+ routePTY, releaseRoute := newBlockingPTY(t)
213+ defer releaseRoute()
214+ d := newTestDaemon(t, newFactorySeq(t, routePTY), stubClock{})
215+ addControlSession(d, "work", "t_work", "p_work")
216+ addNamedMoveDestination(d, "dest", "t_dest", "p_dest")
217+ require.Equal(t, 2, sessionCount(d))
218+
219+ beginTestPurgeAdmission(t, d)
220+
221+ created := make(chan error, 1)
222+ go func() {
223+ _, _, err := d.route(helloResumeCapable(protocol.IntentEphemeral, "", 0), &closeTrackingTransport{})
224+ created <- err
225+ }()
226+ select {
227+ case err := <-created:
228+ t.Fatalf("route bypassed the closed purge admission gate: %v", err)
229+ case <-time.After(50 * time.Millisecond):
230+ }
231+ require.Equal(t, 2, sessionCount(d), "gated creation must not insert a session")
232+
233+ rejected := sendCommand(t, d, protocol.CommandRequest{
234+ Slug: "move-pane", Args: []string{"dest", "t_dest"}, Self: true,
235+ TargetSession: "work", TargetTab: "t_work", TargetPane: "p_work",
236+ })
237+ require.False(t, rejected.OK, "a move during the purge must be rejected")
238+
239+ d.endPurgeAdmission()
240+ require.NoError(t, awaitTestValue(t, created, "route did not resume after purge admission reopened"))
241+ require.Equal(t, 3, sessionCount(d), "admission must reopen for creation")
242+
243+ accepted := sendCommand(t, d, protocol.CommandRequest{
244+ Slug: "move-pane", Args: []string{"dest", "t_dest"}, Self: true,
245+ TargetSession: "work", TargetTab: "t_work", TargetPane: "p_work",
246+ })
247+ require.True(t, accepted.OK, accepted.Text)
248+}
249+
250+// TestPurgeAllLinearizesConcurrentCreate races KillAll against a route create.
251+// The gate drains the create before its exact snapshot or admits it after, so
252+// the create never interleaves with the purge and the daemon stays active.
253+func TestPurgeAllLinearizesConcurrentCreate(t *testing.T) {
254+ sz := domain.Size{Cols: 80, Rows: 24}
255+ for attempt := 0; attempt < 24; attempt++ {
256+ seedPTY, releaseSeed := newBlockingPTY(t)
257+ routePTY, releaseRoute := newBlockingPTY(t)
258+ d := newTestDaemon(t, newFactorySeq(t, seedPTY, routePTY), stubClock{})
259+
260+ seed, err := createSessionForTest(d, "0", true, "/tmp", sz, terminalEnv{}, nil)
261+ require.NoError(t, err)
262+ require.NotNil(t, seed)
263+
264+ start := make(chan struct{})
265+ var wg sync.WaitGroup
266+ wg.Add(2)
267+ var createErr error
268+ go func() {
269+ defer wg.Done()
270+ <-start
271+ _, _, createErr = d.route(helloResumeCapable(protocol.IntentEphemeral, "", 0), &closeTrackingTransport{})
272+ }()
273+ go func() {
274+ defer wg.Done()
275+ <-start
276+ d.purgeAllSessions(protocol.ReasonSessionKilled)
277+ }()
278+ close(start)
279+ wg.Wait()
280+
281+ require.NoError(t, createErr, "attempt %d: creation racing KillAll must be admitted, not rejected", attempt)
282+ count := sessionCount(d)
283+ require.LessOrEqual(t, count, 1, "attempt %d: purge must not leak sessions", attempt)
284+ d.mu.Lock()
285+ closing := d.closing
286+ d.mu.Unlock()
287+ require.False(t, closing, "attempt %d: KillAll must not stop the daemon", attempt)
288+ select {
289+ case <-d.done:
290+ t.Fatalf("attempt %d: KillAll closed done", attempt)
291+ default:
292+ }
293+
294+ killAllSessions(d)
295+ releaseSeed()
296+ releaseRoute()
297+ d.sessWg.Wait()
298+ d.waitNotifies()
299+ }
300+}
301+
302+// TestPurgeAllLinearizesConcurrentRestore races KillAll against a resume of the
303+// only stopped record. The restore is either purged with the set or ordered
304+// after it, and the daemon stays active either way.
305+func TestPurgeAllLinearizesConcurrentRestore(t *testing.T) {
306+ sz := domain.Size{Cols: 80, Rows: 24}
307+ for attempt := 0; attempt < 16; attempt++ {
308+ seedPTY, releaseSeed := newBlockingPTY(t)
309+ restorePTY, releaseRestore := newBlockingPTY(t)
310+ d := newTestDaemon(t, newFactorySeq(t, seedPTY, restorePTY), stubClock{})
311+
312+ seed, err := createSessionForTest(d, "work", false, "/tmp", sz, terminalEnv{}, nil)
313+ require.NoError(t, err)
314+ require.NoError(t, d.killSession(seed, protocol.ReasonServerShutdown, false))
315+ require.Equal(t, 0, sessionCount(d))
316+ d.mu.Lock()
317+ _, stopped := d.inactive["work"]
318+ d.mu.Unlock()
319+ require.True(t, stopped)
320+
321+ start := make(chan struct{})
322+ var wg sync.WaitGroup
323+ wg.Add(2)
324+ var restoreErr error
325+ go func() {
326+ defer wg.Done()
327+ <-start
328+ _, _, restoreErr = d.route(helloResumeCapable(protocol.IntentAttach, "work", 0), &closeTrackingTransport{})
329+ }()
330+ go func() {
331+ defer wg.Done()
332+ <-start
333+ d.purgeAllSessions(protocol.ReasonSessionKilled)
334+ }()
335+ close(start)
336+ wg.Wait()
337+
338+ if restoreErr != nil {
339+ var protoFailure *protoErr
340+ require.ErrorAs(t, restoreErr, &protoFailure)
341+ require.Equal(t, protocol.ErrNoSuchSession, protoFailure.code)
342+ }
343+ require.LessOrEqual(t, sessionCount(d), 1, "attempt %d: purge must not leak sessions", attempt)
344+ d.mu.Lock()
345+ closing := d.closing
346+ d.mu.Unlock()
347+ require.False(t, closing, "attempt %d: KillAll must not stop the daemon", attempt)
348+
349+ killAllSessions(d)
350+ releaseSeed()
351+ releaseRestore()
352+ d.sessWg.Wait()
353+ d.waitNotifies()
354+ }
355+}
356+
357+// TestShutdownAllStillStopsDaemonWithSessionsClientsAndWorkers proves the
358+// explicit daemon-stop operation is unchanged: it removes the session, signals
359+// Serve, marks closing, closes move admission, and cancels the daemon-wide pane
360+// process context.
361+func TestShutdownAllStillStopsDaemonWithSessionsClientsAndWorkers(t *testing.T) {
362+ sz := domain.Size{Cols: 80, Rows: 24}
363+ pty, releasePTY := newBlockingPTY(t)
364+ d := newTestDaemon(t, newFactorySeq(t, pty), stubClock{})
365+
366+ tr, sends, releaseConn := newConn(t, mustHello(protocol.IntentNew, "work", sz))
367+ var hg sync.WaitGroup
368+ hg.Go(func() { d.handleConn(tr) })
369+ awaitFrame(t, sends, "Welcome")
370+ awaitFrame(t, sends, "Output")
371+ require.Equal(t, 1, sessionCount(d))
372+
373+ require.False(t, d.shutdownAll(protocol.ReasonServerShutdown))
374+ require.Equal(t, 0, sessionCount(d))
375+ select {
376+ case <-d.done:
377+ default:
378+ t.Fatal("explicit daemon stop must signal Serve")
379+ }
380+ select {
381+ case <-d.paneProcessCtx.Done():
382+ default:
383+ t.Fatal("explicit daemon stop must cancel the daemon-wide pane process context")
384+ }
385+ d.mu.Lock()
386+ closing := d.closing
387+ d.mu.Unlock()
388+ require.True(t, closing)
389+ d.moveLifecycleMu.Lock()
390+ moveClosing := d.moveLifecycleClosing
391+ d.moveLifecycleMu.Unlock()
392+ require.True(t, moveClosing)
393+
394+ releaseConn()
395+ releasePTY()
396+ hg.Wait()
397+ d.sessWg.Wait()
398+ d.waitNotifies()
399+}
400+
401+// beginTestPurgeAdmission closes the KillAll admission gate exactly as a purge
402+// does once its drain succeeds. It fails the test if the gate could not be
403+// granted (an admitted transition still holding it, shutdown, or cancellation).
404+func beginTestPurgeAdmission(t *testing.T, d *Daemon) {
405+ t.Helper()
406+ if status := d.beginPurgeAdmission(context.Background(), nil); status != purgeAdmissionGranted {
407+ t.Fatalf("test purge admission not granted: status %d", status)
408+ }
409+}
410+
411+// killAllSessions removes every remaining live session without asserting on the
412+// result; it exists to release test PTYs deterministically.
413+func killAllSessions(d *Daemon) {
414+ d.mu.Lock()
415+ sessions := d.sessionsSnapshotLocked()
416+ d.mu.Unlock()
417+ for _, sess := range sessions {
418+ _ = d.killSession(sess, protocol.ReasonSessionKilled, true)
419+ }
420+}
421+
422+// gatedCheckpointRepository blocks LoadCheckpoint until released, so a test can
423+// hold a restoration worker inside its admitted lifespan deterministically and
424+// observe KillAll admission against it.
425+type gatedCheckpointRepository struct {
426+ noOpSnapshotRepository
427+ generation ports.SnapshotGeneration
428+ entered chan struct{}
429+ release chan struct{}
430+ once sync.Once
431+}
432+
433+func newGatedCheckpointRepository(generation ports.SnapshotGeneration) *gatedCheckpointRepository {
434+ return &gatedCheckpointRepository{generation: generation, entered: make(chan struct{}), release: make(chan struct{})}
435+}
436+
437+func (r *gatedCheckpointRepository) LoadCheckpoint(ctx context.Context, _ domain.IncarnationID, _ string, _ ports.CheckpointRef) (ports.SnapshotGeneration, error) {
438+ r.once.Do(func() { close(r.entered) })
439+ select {
440+ case <-r.release:
441+ case <-ctx.Done():
442+ return ports.SnapshotGeneration{}, ctx.Err()
443+ }
444+ return r.generation, nil
445+}
446+
447+// syncedLogBuffer collects log output for assertions without racing the
448+// daemon's asynchronous sends.
449+type syncedLogBuffer struct {
450+ mu sync.Mutex
451+ buf bytes.Buffer
452+}
453+
454+func (b *syncedLogBuffer) Write(p []byte) (int, error) {
455+ b.mu.Lock()
456+ defer b.mu.Unlock()
457+ return b.buf.Write(p)
458+}
459+
460+func (b *syncedLogBuffer) String() string {
461+ b.mu.Lock()
462+ defer b.mu.Unlock()
463+ return b.buf.String()
464+}
465+
466+// TestPurgeAllWaitsForAdmittedRestoreThenPurges is the deterministic gated
467+// checkpoint contract: KillAll blocks while a startup restoration worker holds
468+// purge admission inside its load, then purges the entire exact set once the
469+// restore settles. No phantom live session, stopped authority, or taken name
470+// may survive.
471+func TestPurgeAllWaitsForAdmittedRestoreThenPurges(t *testing.T) {
472+ snapshot := restoreAcceptanceSession(t, "work")
473+ generation := acceptanceGeneration(t, snapshot, 9)
474+ repository := newGatedCheckpointRepository(generation)
475+ pty, releasePTY := newBlockingPTY(t)
476+ defer releasePTY()
477+
478+ d := newTestDaemon(t, newFactorySeq(t, pty), stubClock{})
479+ d.baseEnv = []string{"TERM=xterm-256color"}
480+ checkpoint := domain.CheckpointRef{Generation: generation.Generation, ManifestDigest: snapcodec.ManifestDigest(generation.Manifest)}
481+ record := domain.CatalogueRecord{Name: snapshot.Name, IncarnationID: generation.IncarnationID, Cwd: "/snapshot/cwd", CreatedAt: int64(snapshot.CreatedAt), Committed: &checkpoint}
482+ catalogue := newDurableRecoveryCatalogue([]domain.CatalogueRecord{record})
483+ WithCatalogue(catalogue, []domain.CatalogueRecord{record})(d)
484+ WithSnapshotRepository(repository)(d)
485+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, repository, nil))(d)
486+
487+ restoreFinished := make(chan struct{})
488+ go func() {
489+ defer close(restoreFinished)
490+ d.restoreIncrementalSnapshots(context.Background())
491+ }()
492+ <-repository.entered
493+
494+ purged := make(chan purgeResult, 1)
495+ go func() { purged <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
496+ select {
497+ case result := <-purged:
498+ t.Fatalf("KillAll returned while an admitted restore was still loading: %v", result.summary())
499+ case <-time.After(100 * time.Millisecond):
500+ }
501+
502+ close(repository.release)
503+ <-restoreFinished
504+ result := awaitTestValue(t, purged, "KillAll did not complete after the admitted restore settled")
505+ require.False(t, result.Superseded())
506+ require.False(t, result.Failed(), result.summary())
507+
508+ require.Equal(t, 0, sessionCount(d), "no live authority may survive the purge")
509+ d.mu.Lock()
510+ _, stopped := d.inactive["work"]
511+ closing := d.closing
512+ d.mu.Unlock()
513+ require.False(t, stopped, "no phantom stopped authority may survive the purge")
514+ require.False(t, closing, "KillAll must leave the daemon serving")
515+ select {
516+ case <-d.done:
517+ t.Fatal("KillAll must not signal Serve")
518+ default:
519+ }
520+ _, ok, err := d.catalogue.Record("work")
521+ require.NoError(t, err)
522+ require.False(t, ok, "durable authority must be purged")
523+
524+ // The purged name is reusable by the same daemon.
525+ d.mu.Lock()
526+ taken := d.nameLiveOrStoppedLocked("work")
527+ d.mu.Unlock()
528+ require.False(t, taken, "the purged name must not remain taken")
529+ d.sessWg.Wait()
530+ d.waitNotifies()
531+}
532+
533+// TestRestoreCatalogueAbandonsRecordsSupersededByPurge proves the epoch
534+// revalidation: a restoration that captured records before a completed KillAll
535+// must not resurrect them from the pre-purge record set.
536+func TestRestoreCatalogueAbandonsRecordsSupersededByPurge(t *testing.T) {
537+ record := durableRecoveryRecord(1)
538+ record.Name = "work"
539+ catalogue := newDurableRecoveryCatalogue([]domain.CatalogueRecord{record})
540+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), stubClock{})
541+ WithCatalogue(catalogue, []domain.CatalogueRecord{record})(d)
542+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, noOpSnapshotRepository{}, nil))(d)
543+
544+ epoch := d.purgeEpochSnapshot()
545+ result := d.purgeAllSessions(protocol.ReasonSessionKilled)
546+ require.False(t, result.Failed(), result.summary())
547+ _, ok, err := catalogue.Record("work")
548+ require.NoError(t, err)
549+ require.False(t, ok, "the purge must remove the durable record")
550+
551+ d.restoreCatalogueEpoch(context.Background(), []domain.CatalogueRecord{record}, epoch)
552+ require.Equal(t, 0, sessionCount(d))
553+ d.mu.Lock()
554+ _, stopped := d.inactive["work"]
555+ d.mu.Unlock()
556+ require.False(t, stopped, "a superseded record must not be resurrected as stopped authority")
557+ d.sessWg.Wait()
558+ d.waitNotifies()
559+}
560+
561+// TestEnsureCatalogueRegistryEntryRejectsSupersededRestore proves both the
562+// in-progress gate and the completed-purge epoch are revalidated before a
563+// restore publishes any registry entry.
564+func TestEnsureCatalogueRegistryEntryRejectsSupersededRestore(t *testing.T) {
565+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), stubClock{})
566+ record := durableRecoveryRecord(1)
567+ record.Name = "work"
568+
569+ prePurgeEpoch := d.purgeEpochSnapshot()
570+ beginTestPurgeAdmission(t, d)
571+ done, ok := d.ensureCatalogueRegistryEntry(record, prePurgeEpoch)
572+ require.False(t, ok, "a record must not be admitted while a purge owns the gate")
573+ require.Nil(t, done)
574+ d.endPurgeAdmission()
575+
576+ // A worker that waited out the purge still holds a stale boundary: the
577+ // record set it captured was superseded, so it must stay abandoned.
578+ done, ok = d.ensureCatalogueRegistryEntry(record, prePurgeEpoch)
579+ require.False(t, ok, "a record captured before a completed purge must stay superseded")
580+ require.Nil(t, done)
581+
582+ // A fresh boundary (post-purge) admits the record.
583+ done, ok = d.ensureCatalogueRegistryEntry(record, d.purgeEpochSnapshot())
584+ require.True(t, ok)
585+ require.NotNil(t, done)
586+ d.mu.Lock()
587+ _, exists := d.inactive["work"]
588+ d.mu.Unlock()
589+ require.True(t, exists)
590+}
591+
592+// TestRestoreCatalogueCancellationDrainsAdmission proves the shutdown interlock:
593+// a restoration worker blocked on purge admission observes context
594+// cancellation, drains its job queue without touching the registry, and lets
595+// the producer and the worker join.
596+func TestRestoreCatalogueCancellationDrainsAdmission(t *testing.T) {
597+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), stubClock{})
598+ record := durableRecoveryRecord(1)
599+ record.Name = "work"
600+
601+ ctx, cancel := context.WithCancel(context.Background())
602+ done := make(chan struct{})
603+ beginTestPurgeAdmission(t, d)
604+ go func() {
605+ defer close(done)
606+ d.restoreCatalogueEpoch(ctx, []domain.CatalogueRecord{record}, d.purgeEpochSnapshot())
607+ }()
608+ // The worker is parked on admission; cancellation must release it even
609+ // though the purge gate is still closed.
610+ cancel()
611+ awaitTestValue(t, done, "restoration did not drain after cancellation")
612+ d.endPurgeAdmission()
613+
614+ d.mu.Lock()
615+ _, exists := d.inactive["work"]
616+ d.mu.Unlock()
617+ require.False(t, exists, "a cancelled restoration must not register a stopped entry")
618+}
619+
620+// TestPurgeAllSerializesConcurrentKillAll proves purgeAllMu admits one KillAll
621+// owner at a time: a second concurrent purge cannot enter the critical section
622+// until the first releases it.
623+func TestPurgeAllSerializesConcurrentKillAll(t *testing.T) {
624+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), stubClock{})
625+ entered := make(chan struct{})
626+ releaseFirst := make(chan struct{})
627+ var calls atomic.Int32
628+ d.afterPurgeAdmission = func() {
629+ if calls.Add(1) == 1 {
630+ close(entered)
631+ <-releaseFirst
632+ }
633+ }
634+
635+ first := make(chan purgeResult, 1)
636+ go func() { first <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
637+ awaitTestValue(t, entered, "first purge never entered its critical section")
638+
639+ second := make(chan purgeResult, 1)
640+ go func() { second <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
641+ select {
642+ case <-second:
643+ t.Fatal("a second KillAll entered the critical section while the first still owned it")
644+ case <-time.After(100 * time.Millisecond):
645+ }
646+ close(releaseFirst)
647+ require.False(t, awaitTestValue(t, first, "first purge did not finish").Failed())
648+ require.False(t, awaitTestValue(t, second, "second purge did not run after the first released").Failed())
649+ require.Equal(t, int32(2), calls.Load())
650+}
651+
652+// TestCommandPathPurgeAdmissionMapsToRetryableError proves a command-path
653+// create rejected by the purge gate surfaces a deterministic, retryable result
654+// rather than silently succeeding or reusing a name verdict.
655+func TestCommandPathPurgeAdmissionMapsToRetryableError(t *testing.T) {
656+ routePTY, releaseRoute := newBlockingPTY(t)
657+ defer releaseRoute()
658+ d := newTestDaemon(t, newFactorySeq(t, routePTY), stubClock{})
659+ addControlSession(d, "work", "t_work", "p_work")
660+
661+ beginTestPurgeAdmission(t, d)
662+ result := sendCommand(t, d, protocol.CommandRequest{
663+ Slug: "new-session", Args: []string{"fresh"}, Self: true,
664+ TargetSession: "work", TargetTab: "t_work", TargetPane: "p_work",
665+ })
666+ require.False(t, result.OK)
667+ require.Equal(t, protocol.ErrInternal, result.Code)
668+ require.Equal(t, errPurgeAdmissionClosed.Error(), result.Text)
669+ d.endPurgeAdmission()
670+}
671+
672+// TestRemoteTargetRouteWaitsForPurgeAdmission proves an exact remote-target
673+// handoff is admitted like a local route: it blocks on the closed purge gate,
674+// then proceeds after the gate reopens.
675+func TestRemoteTargetRouteWaitsForPurgeAdmission(t *testing.T) {
676+ sz := domain.Size{Cols: 80, Rows: 24}
677+ pty, releasePTY := newBlockingPTY(t)
678+ defer releasePTY()
679+ d := newTestDaemon(t, newFactorySeq(t, pty), stubClock{})
680+ sess, err := createSessionForTest(d, "work", false, "/tmp", sz, terminalEnv{}, nil)
681+ require.NoError(t, err)
682+ sess.mu.Lock()
683+ incarnation := sess.incarnation
684+ tabID := sess.tabs[0].stableID
685+ sess.mu.Unlock()
686+ target := domain.RemoteSessionTarget{Endpoint: "arch", DisplayOrigin: "arch", LifecycleID: incarnation, SessionName: "work", LiveTabID: domain.TabStableID(tabID)}
687+ tr, _ := newCapturingTransport(t)
688+
689+ beginTestPurgeAdmission(t, d)
690+ result := make(chan error, 1)
691+ go func() {
692+ _, _, routeErr := d.routeWithContext(context.Background(), protocol.Hello{
693+ Version: protocol.Version, Intent: protocol.IntentAttach, Name: "work", Size: sz,
694+ RemoteTarget: &target, EnvironmentPolicy: protocol.EnvironmentPolicyDaemonOwned,
695+ }, tr)
696+ result <- routeErr
697+ }()
698+ select {
699+ case routeErr := <-result:
700+ t.Fatalf("remote target route bypassed the closed purge gate: %v", routeErr)
701+ case <-time.After(50 * time.Millisecond):
702+ }
703+ d.endPurgeAdmission()
704+ require.NoError(t, awaitTestValue(t, result, "remote target route did not resume after the gate reopened"))
705+
706+ _ = d.killSession(sess, protocol.ReasonSessionKilled, true)
707+ releasePTY()
708+ d.sessWg.Wait()
709+ d.waitNotifies()
710+}
711+
712+// TestConcurrentCreateAndExplicitShutdownInterlock restores a real concurrent
713+// create-versus-explicit-shutdown test: whichever side wins the registry lock,
714+// no session leaks, the daemon is irreversibly closing and signaled, and later
715+// Hellos are rejected.
716+func TestConcurrentCreateAndExplicitShutdownInterlock(t *testing.T) {
717+ for attempt := 0; attempt < 24; attempt++ {
718+ pty, releasePTY := newBlockingPTY(t)
719+ d := newTestDaemon(t, newFactorySeq(t, pty), stubClock{})
720+
721+ start := make(chan struct{})
722+ var wg sync.WaitGroup
723+ var createErr error
724+ wg.Add(2)
725+ go func() {
726+ defer wg.Done()
727+ <-start
728+ _, _, createErr = d.route(helloResumeCapable(protocol.IntentEphemeral, "", 0), &closeTrackingTransport{})
729+ }()
730+ go func() {
731+ defer wg.Done()
732+ <-start
733+ d.shutdownAll(protocol.ReasonServerShutdown)
734+ }()
735+ close(start)
736+ wg.Wait()
737+
738+ if createErr != nil {
739+ var protoFailure *protoErr
740+ require.ErrorAs(t, createErr, &protoFailure, "attempt %d", attempt)
741+ require.Equal(t, protocol.ErrServerShutdown, protoFailure.code, "attempt %d", attempt)
742+ }
743+ require.Equal(t, 0, sessionCount(d), "attempt %d: explicit shutdown must not leak a session", attempt)
744+ d.mu.Lock()
745+ closing := d.closing
746+ d.mu.Unlock()
747+ require.True(t, closing, "attempt %d: explicit shutdown must mark closing", attempt)
748+ select {
749+ case <-d.done:
750+ default:
751+ t.Fatalf("attempt %d: explicit shutdown must signal Serve", attempt)
752+ }
753+ _, _, afterErr := d.route(helloResumeCapable(protocol.IntentEphemeral, "", 0), &closeTrackingTransport{})
754+ require.Error(t, afterErr, "attempt %d: a Hello after shutdown must be rejected", attempt)
755+
756+ releasePTY()
757+ d.sessWg.Wait()
758+ d.waitNotifies()
759+ }
760+}
761+
762+// TestPurgeAllSupersededByExplicitShutdownPreservesStopAuthority proves an
763+// explicit daemon stop wins over a KillAll that has not started removing
764+// anything: the stopped authority the stop preserved is left intact.
765+func TestPurgeAllSupersededByExplicitShutdownPreservesStopAuthority(t *testing.T) {
766+ sz := domain.Size{Cols: 80, Rows: 24}
767+ pty, releasePTY := newBlockingPTY(t)
768+ defer releasePTY()
769+ d := newTestDaemon(t, newFactorySeq(t, pty), stubClock{})
770+ catalogue := newDurableRecoveryCatalogue(nil)
771+ WithCatalogue(catalogue, nil)(d)
772+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, noOpSnapshotRepository{}, bytes.NewReader(bytes.Repeat([]byte{0x5a}, 32))))(d)
773+
774+ _, err := createSessionForTest(d, "work", false, "/tmp", sz, terminalEnv{}, nil)
775+ require.NoError(t, err)
776+ require.False(t, d.shutdownAll(protocol.ReasonServerShutdown))
777+ require.Equal(t, 0, sessionCount(d))
778+
779+ result := d.purgeAllSessions(protocol.ReasonSessionKilled)
780+ require.True(t, result.Superseded())
781+ d.mu.Lock()
782+ _, stopped := d.inactive["work"]
783+ d.mu.Unlock()
784+ require.True(t, stopped, "explicit stop must preserve stopped authority")
785+ _, ok, err := catalogue.Record("work")
786+ require.NoError(t, err)
787+ require.True(t, ok, "explicit stop must preserve durable authority")
788+
789+ releasePTY()
790+ d.sessWg.Wait()
791+ d.waitNotifies()
792+}
793+
794+// TestPurgeAllLogsFailedControlSend proves a purge failure that cannot be
795+// delivered is logged, so a client observing EOF cannot silently infer success.
796+func TestPurgeAllLogsFailedControlSend(t *testing.T) {
797+ repository := newSelectivePurgeRepository(map[domain.IncarnationID]error{{2}: errors.New("delete failed")})
798+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), stubClock{})
799+ var logged syncedLogBuffer
800+ d.log = slog.New(slog.NewTextHandler(&logged, nil))
801+ WithSnapshotRepository(repository)(d)
802+ store, _ := newMockStore(t)
803+ WithStore(t, store)(d)
804+
805+ record := domain.CatalogueRecord{Name: "stopped", IncarnationID: domain.IncarnationID{2}, CreatedAt: 7}
806+ require.NoError(t, d.catalogue.Create(record))
807+ d.inactive["stopped"] = inactiveSessionFromRecord(record, protocol.SessionDown, nil)
808+
809+ tr := newMockServerConnection(t)
810+ tr.EXPECT().Send(mock.Anything).Return(errors.New("client gone")).Maybe()
811+ tr.EXPECT().Close().Return(nil).Maybe()
812+ d.handleKill(tr, protocol.Kill{Scope: protocol.KillAll})
813+
814+ require.Contains(t, logged.String(), "kill all failure response")
815+ require.Contains(t, logged.String(), "client gone")
816+}
817+
818+// TestPurgeResultSummaryBoundsWireDetail proves the partial-failure summary is
819+// wire-safe: it reports the total count, only the first N failures, and how
820+// many were omitted.
821+func TestPurgeResultSummaryBoundsWireDetail(t *testing.T) {
822+ result := purgeResult{}
823+ for i := 0; i < purgeSummaryFailureLimit+3; i++ {
824+ result.Failures = append(result.Failures, purgeFailure{Class: purgeRecordLive, Name: fmt.Sprintf("s%02d", i), Err: errors.New("boom")})
825+ }
826+ summary := result.summary()
827+ require.Contains(t, summary, fmt.Sprintf("%d record(s) could not be purged", purgeSummaryFailureLimit+3))
828+ require.Contains(t, summary, "; 3 more omitted")
829+ require.Contains(t, summary, "s00")
830+ require.NotContains(t, summary, "s07", "failures beyond the bound must be omitted from the wire detail")
831+}
832+
833+// TestPurgeAllPurgesSettledRestore proves the other half of the policy: when the
834+// admitted restore settles as a live session before KillAll begins, the purge's
835+// exact snapshot includes it and removes it like any other live session.
836+func TestPurgeAllPurgesSettledRestore(t *testing.T) {
837+ snapshot := restoreAcceptanceSession(t, "work")
838+ generation := acceptanceGeneration(t, snapshot, 4)
839+ repository := newGatedCheckpointRepository(generation)
840+ pty, releasePTY := newBlockingPTY(t)
841+ defer releasePTY()
842+
843+ d := newTestDaemon(t, newFactorySeq(t, pty), stubClock{})
844+ d.baseEnv = []string{"TERM=xterm-256color"}
845+ checkpoint := domain.CheckpointRef{Generation: generation.Generation, ManifestDigest: snapcodec.ManifestDigest(generation.Manifest)}
846+ record := domain.CatalogueRecord{Name: snapshot.Name, IncarnationID: generation.IncarnationID, Cwd: "/snapshot/cwd", CreatedAt: int64(snapshot.CreatedAt), Committed: &checkpoint}
847+ catalogue := newDurableRecoveryCatalogue([]domain.CatalogueRecord{record})
848+ WithCatalogue(catalogue, []domain.CatalogueRecord{record})(d)
849+ WithSnapshotRepository(repository)(d)
850+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, repository, nil))(d)
851+
852+ restoreFinished := make(chan struct{})
853+ go func() {
854+ defer close(restoreFinished)
855+ d.restoreIncrementalSnapshots(context.Background())
856+ }()
857+ <-repository.entered
858+ close(repository.release)
859+ awaitTestValue(t, restoreFinished, "restore did not settle")
860+ require.Equal(t, 1, sessionCount(d), "the settled restore must be live before the purge")
861+
862+ result := d.purgeAllSessions(protocol.ReasonSessionKilled)
863+ require.False(t, result.Superseded())
864+ require.False(t, result.Failed(), result.summary())
865+ require.Equal(t, 0, sessionCount(d), "the settled restore must be purged")
866+ _, ok, err := d.catalogue.Record("work")
867+ require.NoError(t, err)
868+ require.False(t, ok)
869+ d.sessWg.Wait()
870+ d.waitNotifies()
871+}
872+
873+// TestPurgeAllConcurrentWithExplicitShutdown proves the KillAll-versus-KillDaemon
874+// interlock terminates safely: whichever side wins, the daemon irreversibly
875+// stops, no live session leaks, and Serve is signaled.
876+func TestPurgeAllConcurrentWithExplicitShutdown(t *testing.T) {
877+ sz := domain.Size{Cols: 80, Rows: 24}
878+ for attempt := 0; attempt < 16; attempt++ {
879+ pty, releasePTY := newBlockingPTY(t)
880+ d := newTestDaemon(t, newFactorySeq(t, pty), stubClock{})
881+ _, err := createSessionForTest(d, "work", false, "/tmp", sz, terminalEnv{}, nil)
882+ require.NoError(t, err)
883+
884+ start := make(chan struct{})
885+ var wg sync.WaitGroup
886+ var result purgeResult
887+ wg.Add(2)
888+ go func() {
889+ defer wg.Done()
890+ <-start
891+ result = d.purgeAllSessions(protocol.ReasonSessionKilled)
892+ }()
893+ go func() {
894+ defer wg.Done()
895+ <-start
896+ d.shutdownAll(protocol.ReasonServerShutdown)
897+ }()
898+ close(start)
899+ wg.Wait()
900+
901+ require.Equal(t, 0, sessionCount(d), "attempt %d: no live session may survive the race", attempt)
902+ d.mu.Lock()
903+ closing := d.closing
904+ d.mu.Unlock()
905+ require.True(t, closing, "attempt %d: the explicit stop must win", attempt)
906+ select {
907+ case <-d.done:
908+ default:
909+ t.Fatalf("attempt %d: the explicit stop must signal Serve", attempt)
910+ }
911+ if !result.Superseded() {
912+ require.False(t, result.Failed(), "attempt %d: %s", attempt, result.summary())
913+ }
914+ releasePTY()
915+ d.sessWg.Wait()
916+ d.waitNotifies()
917+ }
918+}
919+
920+// cancellationIgnoringRepository blocks LoadCheckpoint until released and
921+// deliberately ignores context cancellation, modelling a repository call that
922+// never returns. It proves the purge admission drain and daemon shutdown stay
923+// bounded instead of waiting on an uncooperative transition.
924+type cancellationIgnoringRepository struct {
925+ noOpSnapshotRepository
926+
927+ generation ports.SnapshotGeneration
928+ entered chan struct{}
929+ release chan struct{}
930+ once sync.Once
931+}
932+
933+func newCancellationIgnoringRepository(generation ports.SnapshotGeneration) *cancellationIgnoringRepository {
934+ return &cancellationIgnoringRepository{
935+ generation: generation,
936+ entered: make(chan struct{}),
937+ release: make(chan struct{}),
938+ }
939+}
940+
941+func (r *cancellationIgnoringRepository) LoadCheckpoint(context.Context, domain.IncarnationID, string, ports.CheckpointRef) (ports.SnapshotGeneration, error) {
942+ r.once.Do(func() { close(r.entered) })
943+ <-r.release
944+ return r.generation, nil
945+}
946+
947+// awaitPurgeGateClosed waits until a purge has closed the transient admission
948+// gate, so a test can interleave another transition deterministically without
949+// polling on a real timer.
950+func awaitPurgeGateClosed(t *testing.T, d *Daemon) {
951+ t.Helper()
952+ deadline := time.NewTimer(testWaitTimeout)
953+ defer deadline.Stop()
954+ for {
955+ d.mu.Lock()
956+ closed := d.purgeAdmissionClosing
957+ changed := d.purgeAdmissionChangeLocked()
958+ d.mu.Unlock()
959+ if closed {
960+ return
961+ }
962+ select {
963+ case <-changed:
964+ case <-deadline.C:
965+ t.Fatal("purge never closed admission")
966+ }
967+ }
968+}
969+
970+// requirePurgeAdmissionBalanced asserts a completed KillAll left no outstanding
971+// create/restore reservation and re-opened the transient admission gate, so the
972+// daemon keeps admitting transitions after the purge.
973+func requirePurgeAdmissionBalanced(t *testing.T, d *Daemon) {
974+ t.Helper()
975+ d.mu.Lock()
976+ active := d.purgeAdmissionActive
977+ closing := d.purgeAdmissionClosing
978+ d.mu.Unlock()
979+ require.Zero(t, active, "a completed purge must leave no outstanding admission reservation")
980+ require.False(t, closing, "a completed purge must re-open purge admission")
981+}
982+
983+// TestRestoreCatalogueCaptureOrdersAfterPurgeNotResurrect reproduces the
984+// capture-during-purge race deterministically: while a purge owns the admission
985+// gate (closed but not yet snapshotted), a catalogue capture must block on the
986+// gate. It can then only observe the post-purge catalogue, so it cannot
987+// resurrect the record the purge removed with a boundary that looks current.
988+func TestRestoreCatalogueCaptureOrdersAfterPurgeNotResurrect(t *testing.T) {
989+ record := durableRecoveryRecord(1)
990+ record.Name = "work"
991+ catalogue := newDurableRecoveryCatalogue([]domain.CatalogueRecord{record})
992+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), stubClock{})
993+ WithCatalogue(catalogue, []domain.CatalogueRecord{record})(d)
994+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, noOpSnapshotRepository{}, nil))(d)
995+
996+ // Hold one admitted transition so the purge closes the gate and parks in its
997+ // drain: the exact interleaving point where a capture previously slipped in.
998+ require.NoError(t, d.acquirePurgeAdmission(context.Background()))
999+ purged := make(chan purgeResult, 1)
1000+ go func() { purged <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
1001+ awaitPurgeGateClosed(t, d)
1002+
1003+ captured := make(chan struct{})
1004+ go func() {
1005+ defer close(captured)
1006+ d.restoreIncrementalSnapshots(context.Background())
1007+ }()
1008+ select {
1009+ case <-captured:
1010+ t.Fatal("catalogue capture bypassed the closed purge gate")
1011+ case <-time.After(50 * time.Millisecond):
1012+ }
1013+ d.mu.Lock()
1014+ _, published := d.inactive["work"]
1015+ d.mu.Unlock()
1016+ require.False(t, published, "no registry entry may be published while the purge owns the gate")
1017+
1018+ d.releasePurgeAdmission()
1019+ require.False(t, awaitTestValue(t, purged, "purge did not complete after admission drained").Failed())
1020+ awaitTestValue(t, captured, "capture did not finish after the purge reopened admission")
1021+
1022+ require.Equal(t, 0, sessionCount(d))
1023+ d.mu.Lock()
1024+ _, stopped := d.inactive["work"]
1025+ d.mu.Unlock()
1026+ require.False(t, stopped, "a capture ordered after the purge must not resurrect the record")
1027+ _, ok, err := catalogue.Record("work")
1028+ require.NoError(t, err)
1029+ require.False(t, ok, "the purge must remove the durable record")
1030+ d.sessWg.Wait()
1031+ d.waitNotifies()
1032+}
1033+
1034+// TestPurgeAllAdmissionDrainBoundedWithUncooperativeRepository proves a KillAll
1035+// whose drain is blocked by a repository call that ignores cancellation returns
1036+// a structured failure at its bounded budget, leaves the daemon serving, and
1037+// does not prevent a later explicit shutdown from completing.
1038+func TestPurgeAllAdmissionDrainBoundedWithUncooperativeRepository(t *testing.T) {
1039+ snapshot := restoreAcceptanceSession(t, "work")
1040+ generation := acceptanceGeneration(t, snapshot, 5)
1041+ repository := newCancellationIgnoringRepository(generation)
1042+ clock := &signalClock{timers: make(chan *signalTimer, 8)}
1043+ pty, releasePTY := newBlockingPTY(t)
1044+ defer releasePTY()
1045+
1046+ d := newTestDaemon(t, newFactorySeq(t, pty), clock)
1047+ d.baseEnv = []string{"TERM=xterm-256color"}
1048+ checkpoint := domain.CheckpointRef{Generation: generation.Generation, ManifestDigest: snapcodec.ManifestDigest(generation.Manifest)}
1049+ record := domain.CatalogueRecord{Name: snapshot.Name, IncarnationID: generation.IncarnationID, Cwd: "/snapshot/cwd", CreatedAt: int64(snapshot.CreatedAt), Committed: &checkpoint}
1050+ catalogue := newDurableRecoveryCatalogue([]domain.CatalogueRecord{record})
1051+ WithCatalogue(catalogue, []domain.CatalogueRecord{record})(d)
1052+ WithSnapshotRepository(repository)(d)
1053+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, repository, nil))(d)
1054+
1055+ restoreFinished := make(chan struct{})
1056+ go func() {
1057+ defer close(restoreFinished)
1058+ d.restoreIncrementalSnapshots(context.Background())
1059+ }()
1060+ <-repository.entered
1061+
1062+ purged := make(chan purgeResult, 1)
1063+ go func() { purged <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
1064+
1065+ // The purge is parked in its admission drain; fire the bounded deadline.
1066+ timer := awaitTestValue(t, clock.timers, "purge did not arm its bounded admission deadline")
1067+ timer.ch <- time.Now()
1068+
1069+ result := awaitTestValue(t, purged, "KillAll did not return after its bounded drain")
1070+ require.True(t, result.Failed(), "a bounded drain timeout must be a structured failure")
1071+ require.False(t, result.Superseded())
1072+ require.Contains(t, result.summary(), "did not drain")
1073+
1074+ d.mu.Lock()
1075+ _, stopped := d.inactive["work"]
1076+ closing := d.closing
1077+ d.mu.Unlock()
1078+ require.True(t, stopped, "the in-flight record must stay as stopped authority")
1079+ require.False(t, closing, "a bounded purge abort must leave the daemon serving")
1080+ _, ok, err := catalogue.Record("work")
1081+ require.NoError(t, err)
1082+ require.True(t, ok, "an aborted purge must not remove the durable record")
1083+
1084+ // An explicit stop owns teardown and completes even though the repository
1085+ // still refuses to observe cancellation.
1086+ shutdownDone := make(chan struct{})
1087+ go func() {
1088+ defer close(shutdownDone)
1089+ d.shutdownAll(protocol.ReasonServerShutdown)
1090+ }()
1091+ awaitTestValue(t, shutdownDone, "explicit shutdown did not complete with a stuck repository")
1092+ select {
1093+ case <-d.done:
1094+ default:
1095+ t.Fatal("explicit shutdown must signal Serve")
1096+ }
1097+
1098+ close(repository.release)
1099+ awaitTestValue(t, restoreFinished, "restore did not settle after the repository released")
1100+ d.sessWg.Wait()
1101+ d.waitNotifies()
1102+}
1103+
1104+// TestPurgeAllAdmissionDrainDefersToExplicitShutdown proves a KillAll parked in
1105+// its admission drain is woken by an explicit stop and defers immediately,
1106+// rather than waiting out its bounded timeout. This is what keeps Serve from
1107+// wedging behind a purge when an admitted transition refuses to drain.
1108+func TestPurgeAllAdmissionDrainDefersToExplicitShutdown(t *testing.T) {
1109+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), stubClock{})
1110+ require.NoError(t, d.acquirePurgeAdmission(context.Background()))
1111+
1112+ purged := make(chan purgeResult, 1)
1113+ go func() { purged <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
1114+ awaitPurgeGateClosed(t, d)
1115+
1116+ d.shutdownAll(protocol.ReasonServerShutdown)
1117+
1118+ result := awaitTestValue(t, purged, "a parked purge was not woken by the explicit stop")
1119+ require.True(t, result.Superseded(), "a parked purge must defer to the explicit stop")
1120+ require.False(t, result.Failed())
1121+ select {
1122+ case <-d.done:
1123+ default:
1124+ t.Fatal("the explicit stop must signal Serve")
1125+ }
1126+ d.releasePurgeAdmission()
1127+}
1128+
1129+// TestPurgeAllDefersToExplicitShutdownPerUnit proves explicit stop wins per
1130+// unit: once a shutdown begins mid-purge, no further unit is removed, so the
1131+// stop can preserve the not-yet-owned lifecycle. A unit whose teardown the
1132+// purge already owns keeps purge authority and completes as a purge (its
1133+// stopped authority is removed), while the not-yet-owned unit stays live for
1134+// the stop to preserve.
1135+func TestPurgeAllDefersToExplicitShutdownPerUnit(t *testing.T) {
1136+ repository := newSelectivePurgeRepository(nil)
1137+ var d *Daemon
1138+ repository.onDelete = func() {
1139+ d.mu.Lock()
1140+ d.closing = true
1141+ d.signalPurgeAdmissionChangedLocked()
1142+ d.mu.Unlock()
1143+ }
1144+ d = newTestDaemon(t, portsmocks.NewMockPTYFactory(t), stubClock{})
1145+ catalogue := newDurableRecoveryCatalogue(nil)
1146+ WithCatalogue(catalogue, nil)(d)
1147+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, repository, nil))(d)
1148+
1149+ for _, name := range []string{"a", "b"} {
1150+ live := newSnapshotTestSession(t, name, false, "/"+name)
1151+ require.NoError(t, catalogue.Create(live.persistRecordLocked(live.createdAt)))
1152+ livePTY, ok := live.tabs[0].panes["pane-1"].pty.(*portsmocks.MockPTY)
1153+ require.True(t, ok)
1154+ livePTY.EXPECT().Close().Return(nil).Maybe()
1155+ d.sessions[live.id] = live
1156+ }
1157+
1158+ result := d.purgeAllSessions(protocol.ReasonSessionKilled)
1159+ require.True(t, result.Superseded(), "an explicit stop that begins mid-purge must win")
1160+ require.Len(t, repository.calls, 1, "only the purge-owned unit may be removed before the stop wins")
1161+
1162+ d.mu.Lock()
1163+ remaining := len(d.sessions)
1164+ var survivor string
1165+ for _, sess := range d.sessions {
1166+ survivor = sess.name
1167+ }
1168+ _, aStopped := d.inactive["a"]
1169+ _, bStopped := d.inactive["b"]
1170+ d.mu.Unlock()
1171+ require.Equal(t, 1, remaining, "exactly the not-yet-owned unit must survive the purge")
1172+ require.NotEmpty(t, survivor)
1173+ require.False(t, aStopped || bStopped, "the purge-owned unit must not retain stopped authority")
1174+
1175+ // The explicit stop now owns the surviving session and preserves it as
1176+ // stopped authority instead of purging it.
1177+ d.shutdownAll(protocol.ReasonServerShutdown)
1178+ require.Equal(t, 0, sessionCount(d))
1179+ d.mu.Lock()
1180+ _, survivorStopped := d.inactive[survivor]
1181+ d.mu.Unlock()
1182+ require.True(t, survivorStopped, "the explicit stop must preserve the surviving session's authority")
1183+ d.sessWg.Wait()
1184+ d.waitNotifies()
1185+}
1186+
1187+// occupySessionTeardown marks sess as owned by another lifecycle path without
1188+// releasing it, so a bounded kill cannot acquire teardown ownership. The owning
1189+// test releases it with finishTeardown. It models a slow unit parked exactly at
1190+// the destructive boundary: the gate is frozen and teardown ownership is the
1191+// only thing the deadline must outlast.
1192+func occupySessionTeardown(sess *session) {
1193+ sess.teardownMu.Lock()
1194+ sess.teardownActive = true
1195+ sess.teardownDone = make(chan struct{})
1196+ sess.teardownMu.Unlock()
1197+}
1198+
1199+// seedPurgeableLiveSession inserts a named live session with durable authority
1200+// and a closed-safe PTY, then parks its teardown ownership so a purge must reach
1201+// the destructive boundary before it can remove the unit.
1202+func seedPurgeableLiveSession(t *testing.T, d *Daemon, catalogue *durableRecoveryCatalogue, name string) *session {
1203+ t.Helper()
1204+ live := newSnapshotTestSession(t, name, false, "/"+name)
1205+ require.NoError(t, catalogue.Create(live.persistRecordLocked(live.createdAt)))
1206+ livePTY, ok := live.tabs[0].panes["pane-1"].pty.(*portsmocks.MockPTY)
1207+ require.True(t, ok)
1208+ livePTY.EXPECT().Close().Return(nil).Maybe()
1209+ d.sessions[live.id] = live
1210+ occupySessionTeardown(live)
1211+ return live
1212+}
1213+
1214+// TestPurgeAllLiveDeadlineSkipIsTypedAndRetryable proves the destructive
1215+// boundary is observable: a live unit whose teardown ownership the shared live
1216+// budget outlasts is reported as a typed live failure with its exact name, is
1217+// left completely live with its durable authority intact, and is removed by a
1218+// later purge once teardown ownership is available. The admission budget and the
1219+// live budget are separate deadlines, so a fresh live budget is armed when the
1220+// live phase begins.
1221+func TestPurgeAllLiveDeadlineSkipIsTypedAndRetryable(t *testing.T) {
1222+ clock := &signalClock{timers: make(chan *signalTimer, 64)}
1223+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), clock)
1224+ catalogue := newDurableRecoveryCatalogue(nil)
1225+ WithCatalogue(catalogue, nil)(d)
1226+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, noOpSnapshotRepository{}, nil))(d)
1227+
1228+ live := seedPurgeableLiveSession(t, d, catalogue, "work")
1229+
1230+ purged := make(chan purgeResult, 1)
1231+ go func() { purged <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
1232+
1233+ admission := awaitTestValue(t, clock.timers, "purge did not arm its admission deadline")
1234+ liveBudget := awaitTestValue(t, clock.timers, "purge did not arm a separate live-teardown budget")
1235+ require.NotSame(t, admission, liveBudget, "live teardown must spend its own budget, not the admission drain's")
1236+ liveBudget.ch <- time.Now()
1237+
1238+ result := awaitTestValue(t, purged, "purge did not return after the live budget expired")
1239+ require.True(t, result.Failed(), result.summary())
1240+ require.False(t, result.Superseded())
1241+ require.Len(t, result.Failures, 1)
1242+ require.Equal(t, purgeRecordLive, result.Failures[0].Class)
1243+ require.Equal(t, "work", result.Failures[0].Name, "a skipped unit must keep its exact name")
1244+ require.ErrorIs(t, result.Failures[0].Err, errSessionKillDeadline)
1245+
1246+ require.Equal(t, 1, sessionCount(d), "a skipped live unit must stay live")
1247+ d.mu.Lock()
1248+ taken := d.nameLiveOrStoppedLocked("work")
1249+ _, stopped := d.inactive["work"]
1250+ d.mu.Unlock()
1251+ require.True(t, taken, "the skipped name must remain taken and accurately reported")
1252+ require.False(t, stopped, "a skipped live unit must not leave phantom stopped authority")
1253+ requirePurgeAdmissionBalanced(t, d)
1254+ _, ok, err := catalogue.Record("work")
1255+ require.NoError(t, err)
1256+ require.True(t, ok, "a skipped live unit must keep its durable authority for the retry")
1257+
1258+ // The retry has full destructive ownership and must purge the unit exactly.
1259+ live.finishTeardown()
1260+ retry := d.purgeAllSessions(protocol.ReasonSessionKilled)
1261+ require.False(t, retry.Failed(), retry.summary())
1262+ require.False(t, retry.Superseded())
1263+ require.Equal(t, 0, sessionCount(d), "the retried purge must remove the unit")
1264+ d.mu.Lock()
1265+ taken = d.nameLiveOrStoppedLocked("work")
1266+ d.mu.Unlock()
1267+ require.False(t, taken, "the retried purge must free the purged name")
1268+ _, ok, err = catalogue.Record("work")
1269+ require.NoError(t, err)
1270+ require.False(t, ok, "the retried purge must remove durable authority")
1271+ d.sessWg.Wait()
1272+ d.waitNotifies()
1273+}
1274+
1275+// TestPurgeAllLiveBudgetSkipsEveryRemainingSlowUnit defines the N-unit behavior:
1276+// one live budget is shared by every live unit. A single expiry outlasts all N
1277+// parked units, so each remaining unit is skipped, reported as its own typed
1278+// live failure with its exact name, and left live; no unit is silently reported
1279+// as purged.
1280+func TestPurgeAllLiveBudgetSkipsEveryRemainingSlowUnit(t *testing.T) {
1281+ const units = 3
1282+ clock := &signalClock{timers: make(chan *signalTimer, 64)}
1283+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), clock)
1284+ catalogue := newDurableRecoveryCatalogue(nil)
1285+ WithCatalogue(catalogue, nil)(d)
1286+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, noOpSnapshotRepository{}, nil))(d)
1287+
1288+ names := make([]string, 0, units)
1289+ for i := range units {
1290+ name := fmt.Sprintf("slow-%d", i)
1291+ names = append(names, name)
1292+ seedPurgeableLiveSession(t, d, catalogue, name)
1293+ }
1294+ require.Equal(t, units, sessionCount(d))
1295+
1296+ purged := make(chan purgeResult, 1)
1297+ go func() { purged <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
1298+
1299+ _ = awaitTestValue(t, clock.timers, "purge did not arm its admission deadline")
1300+ liveBudget := awaitTestValue(t, clock.timers, "purge did not arm a separate live-teardown budget")
1301+ liveBudget.ch <- time.Now()
1302+
1303+ result := awaitTestValue(t, purged, "purge did not return after the shared live budget expired")
1304+ require.True(t, result.Failed(), result.summary())
1305+ require.False(t, result.Superseded())
1306+ require.Len(t, result.Failures, units, "every remaining unit must be reported")
1307+ for i, failure := range result.Failures {
1308+ require.Equal(t, purgeRecordLive, failure.Class)
1309+ require.Equal(t, names[i], failure.Name)
1310+ require.ErrorIs(t, failure.Err, errSessionKillDeadline)
1311+ }
1312+ require.Equal(t, units, sessionCount(d), "no unit may be removed after the shared live budget expired")
1313+ for _, name := range names {
1314+ _, ok, err := catalogue.Record(name)
1315+ require.NoError(t, err)
1316+ require.True(t, ok, "a skipped unit must keep its durable authority")
1317+ }
1318+
1319+ d.mu.Lock()
1320+ sessions := d.sessionsSnapshotLocked()
1321+ d.mu.Unlock()
1322+ for _, sess := range sessions {
1323+ sess.finishTeardown()
1324+ }
1325+ for _, sess := range sessions {
1326+ require.NoError(t, d.killSession(sess, protocol.ReasonSessionKilled, true))
1327+ }
1328+ require.Equal(t, 0, sessionCount(d))
1329+ d.sessWg.Wait()
1330+ d.waitNotifies()
1331+}
1332+
1333+// TestTerminateAllForShutdownReportsDeadlineSkippedSession proves the shutdown
1334+// verdict: a session that a bounded shutdown deadline skips before teardown
1335+// ownership is reported as an incomplete checkpoint instead of silently
1336+// counting as a completed stop. A later deadline-free pass completes the stop
1337+// once the competing owner releases teardown.
1338+func TestTerminateAllForShutdownReportsDeadlineSkippedSession(t *testing.T) {
1339+ clock := &signalClock{timers: make(chan *signalTimer, 64)}
1340+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), clock)
1341+
1342+ live := newSnapshotTestSession(t, "0", true, "/tmp")
1343+ pty, ok := live.tabs[0].panes["pane-1"].pty.(*portsmocks.MockPTY)
1344+ require.True(t, ok)
1345+ pty.EXPECT().Close().Return(nil).Maybe()
1346+ d.sessions[live.id] = live
1347+ occupySessionTeardown(live)
1348+
1349+ deadline := newSnapshotShutdownDeadline(clock)
1350+ defer deadline.stop()
1351+ timer := awaitTestValue(t, clock.timers, "shutdown deadline did not arm")
1352+ timer.ch <- time.Now()
1353+
1354+ require.True(t, d.terminateAllForShutdown(protocol.ReasonServerShutdown, deadline),
1355+ "a session skipped at the teardown deadline must be reported as an incomplete checkpoint")
1356+ require.Equal(t, 1, sessionCount(d), "the skipped session must remain live")
1357+
1358+ live.finishTeardown()
1359+ require.False(t, d.terminateAllForShutdown(protocol.ReasonServerShutdown, nil))
1360+ require.Equal(t, 0, sessionCount(d), "a deadline-free pass must complete the stop")
1361+ select {
1362+ case <-d.done:
1363+ default:
1364+ t.Fatal("the explicit stop must signal Serve")
1365+ }
1366+ d.sessWg.Wait()
1367+ d.waitNotifies()
1368+}
1369+
1370+// TestPurgeAdmissionDeadlineDefersToConcurrentExplicitStop proves the drain
1371+// deadline select rechecks closing: when an explicit stop begins as the drain
1372+// budget expires, the purge reports supersession instead of a drain timeout and
1373+// removes nothing, so the stop keeps ownership of every lifecycle.
1374+func TestPurgeAdmissionDeadlineDefersToConcurrentExplicitStop(t *testing.T) {
1375+ clock := &signalClock{timers: make(chan *signalTimer, 64)}
1376+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), clock)
1377+ catalogue := newDurableRecoveryCatalogue(nil)
1378+ WithCatalogue(catalogue, nil)(d)
1379+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, noOpSnapshotRepository{}, nil))(d)
1380+
1381+ live := seedPurgeableLiveSession(t, d, catalogue, "work")
1382+ // Hold one admitted transition so the purge parks in its bounded drain.
1383+ require.NoError(t, d.acquirePurgeAdmission(context.Background()))
1384+
1385+ purged := make(chan purgeResult, 1)
1386+ go func() { purged <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
1387+ awaitPurgeGateClosed(t, d)
1388+
1389+ // The stop begins exactly as the budget expires and is deliberately not
1390+ // signalled through the admission change channel, so the deadline is the only
1391+ // select case that can fire.
1392+ d.mu.Lock()
1393+ d.closing = true
1394+ d.mu.Unlock()
1395+ admission := awaitTestValue(t, clock.timers, "purge did not arm its admission deadline")
1396+ admission.ch <- time.Now()
1397+
1398+ result := awaitTestValue(t, purged, "purge did not return after the drain budget expired")
1399+ require.True(t, result.Superseded(), "an explicit stop concurrent with the deadline must win")
1400+ require.False(t, result.Failed())
1401+ require.Equal(t, 1, sessionCount(d), "a superseded purge must remove nothing")
1402+
1403+ d.mu.Lock()
1404+ d.closing = false
1405+ d.mu.Unlock()
1406+ d.releasePurgeAdmission()
1407+ live.finishTeardown()
1408+ require.False(t, d.purgeAllSessions(protocol.ReasonSessionKilled).Failed())
1409+ require.Equal(t, 0, sessionCount(d))
1410+ d.sessWg.Wait()
1411+ d.waitNotifies()
1412+}
1413+
1414+// TestPurgeLiveDeadlineDefersToConcurrentExplicitStop proves the same recheck at
1415+// the destructive boundary: a live unit whose budget expires as a stop begins is
1416+// classified as superseded rather than a teardown failure, and it stays live for
1417+// the stop to preserve.
1418+func TestPurgeLiveDeadlineDefersToConcurrentExplicitStop(t *testing.T) {
1419+ clock := &signalClock{timers: make(chan *signalTimer, 64)}
1420+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), clock)
1421+ catalogue := newDurableRecoveryCatalogue(nil)
1422+ WithCatalogue(catalogue, nil)(d)
1423+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, noOpSnapshotRepository{}, nil))(d)
1424+
1425+ live := seedPurgeableLiveSession(t, d, catalogue, "work")
1426+
1427+ purged := make(chan purgeResult, 1)
1428+ go func() { purged <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
1429+ _ = awaitTestValue(t, clock.timers, "purge did not arm its admission deadline")
1430+ liveBudget := awaitTestValue(t, clock.timers, "purge did not arm a separate live-teardown budget")
1431+
1432+ d.mu.Lock()
1433+ d.closing = true
1434+ d.mu.Unlock()
1435+ liveBudget.ch <- time.Now()
1436+
1437+ result := awaitTestValue(t, purged, "purge did not return after the live budget expired")
1438+ require.True(t, result.Superseded(), "the concurrent explicit stop must win over the live deadline")
1439+ require.False(t, result.Failed())
1440+ require.Equal(t, 1, sessionCount(d), "a superseded live unit must remain live")
1441+
1442+ d.mu.Lock()
1443+ d.closing = false
1444+ d.mu.Unlock()
1445+ live.finishTeardown()
1446+ require.False(t, d.purgeAllSessions(protocol.ReasonSessionKilled).Failed())
1447+ require.Equal(t, 0, sessionCount(d))
1448+ d.sessWg.Wait()
1449+ d.waitNotifies()
1450+}
1451+
1452+// blockingPurgeDeleteRepository blocks its first incarnation deletion until it
1453+// is released and ignores context cancellation, so a bounded sweep can only
1454+// finish by stopping its wait on the uncooperative delete.
1455+type blockingPurgeDeleteRepository struct {
1456+ noOpSnapshotRepository
1457+
1458+ entered chan struct{}
1459+ release chan struct{}
1460+ once sync.Once
1461+}
1462+
1463+func newBlockingPurgeDeleteRepository() *blockingPurgeDeleteRepository {
1464+ return &blockingPurgeDeleteRepository{entered: make(chan struct{}), release: make(chan struct{})}
1465+}
1466+
1467+func (r *blockingPurgeDeleteRepository) DeleteIncarnation(context.Context, domain.IncarnationID) error {
1468+ r.once.Do(func() { close(r.entered) })
1469+ <-r.release
1470+ return nil
1471+}
1472+
1473+// seedStoppedPurgeRecords publishes one stopped and one broken registry record
1474+// with durable authority, so a stopped/broken sweep has real work to bound.
1475+func seedStoppedPurgeRecords(t *testing.T, d *Daemon, catalogue *durableRecoveryCatalogue) {
1476+ t.Helper()
1477+ stopped := domain.CatalogueRecord{Name: "stopped", IncarnationID: domain.IncarnationID{2}, CreatedAt: 7}
1478+ require.NoError(t, catalogue.Create(stopped))
1479+ d.inactive["stopped"] = inactiveSessionFromRecord(stopped, protocol.SessionDown, nil)
1480+ broken := domain.CatalogueRecord{Name: "broken", IncarnationID: domain.IncarnationID{3}, CreatedAt: 8}
1481+ require.NoError(t, catalogue.Create(broken))
1482+ d.inactive["broken"] = inactiveSessionFromRecord(broken, protocol.SessionBroken, nil)
1483+}
1484+
1485+// TestPurgeAllParticipantChangeRetriesOnceAndPurgesExactly proves a participant
1486+// change that races a live purge is never counted as success on its own: the
1487+// first kill aborts with the participants-changed sentinel, the purge retries
1488+// once against a fresh snapshot, and only that retry removes the exact unit.
1489+// Both a detach and a resumed attachment incarnation (a token resume) are
1490+// covered.
1491+func TestPurgeAllParticipantChangeRetriesOnceAndPurgesExactly(t *testing.T) {
1492+ tests := []struct {
1493+ name string
1494+ change func(*Daemon, *session, *attachedClient)
1495+ }{
1496+ {
1497+ name: "detach",
1498+ change: func(d *Daemon, sess *session, ac *attachedClient) {
1499+ require.True(t, d.detachIfCurrentTransport(sess, ac, ac.transportSnapshot()))
1500+ },
1501+ },
1502+ {
1503+ name: "token resume",
1504+ change: func(d *Daemon, sess *session, ac *attachedClient) {
1505+ fresh := &closeTrackingTransport{}
1506+ ac.replaceTransport(fresh)
1507+ ac.installTestAttachmentCapability(sess.captureAttachmentCapability(ac, fresh))
1508+ },
1509+ },
1510+ }
1511+ for _, test := range tests {
1512+ t.Run(test.name, func(t *testing.T) {
1513+ d, sess, ac, _ := newManualSessionWithPTYs(t, newQuietPTY())
1514+ catalogue := newDurableRecoveryCatalogue(nil)
1515+ WithCatalogue(catalogue, nil)(d)
1516+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, noOpSnapshotRepository{}, nil))(d)
1517+ require.NoError(t, catalogue.Create(sess.persistRecordLocked(1)))
1518+
1519+ attempts := 0
1520+ d.afterAttachmentEffectParticipantsSnapshotted = func(action string, _ []*attachedClient) {
1521+ if action != "" {
1522+ return
1523+ }
1524+ attempts++
1525+ if attempts == 1 {
1526+ test.change(d, sess, ac)
1527+ }
1528+ }
1529+ t.Cleanup(func() { d.afterAttachmentEffectParticipantsSnapshotted = nil })
1530+
1531+ result := d.purgeAllSessions(protocol.ReasonSessionKilled)
1532+ require.False(t, result.Failed(), result.summary())
1533+ require.False(t, result.Superseded())
1534+ require.Equal(t, 2, attempts, "the aborted first attempt must be retried once")
1535+ require.Equal(t, 0, sessionCount(d), "only the fresh-snapshot retry may remove the unit")
1536+ d.mu.Lock()
1537+ taken := d.nameLiveOrStoppedLocked("work")
1538+ _, stopped := d.inactive["work"]
1539+ d.mu.Unlock()
1540+ require.False(t, taken, "the retried purge must free the purged name")
1541+ require.False(t, stopped, "a purged unit must not leave phantom stopped authority")
1542+ _, ok, err := catalogue.Record("work")
1543+ require.NoError(t, err)
1544+ require.False(t, ok, "the retried purge must remove durable authority")
1545+ d.sessWg.Wait()
1546+ d.waitNotifies()
1547+ })
1548+ }
1549+}
1550+
1551+// TestPurgeAllParticipantChurnIsTypedNotFalseSuccess proves the no-false-success
1552+// contract: a live unit whose participants keep changing is reported as a typed
1553+// live failure with its exact name, keeps its live identity and durable
1554+// authority, and is removed only when the churn stops. The churn alternates a
1555+// detach with a resumed membership, so every attempt's snapshot is stale.
1556+func TestPurgeAllParticipantChurnIsTypedNotFalseSuccess(t *testing.T) {
1557+ d, sess, ac, _ := newManualSessionWithPTYs(t, newQuietPTY())
1558+ catalogue := newDurableRecoveryCatalogue(nil)
1559+ WithCatalogue(catalogue, nil)(d)
1560+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, noOpSnapshotRepository{}, nil))(d)
1561+ require.NoError(t, catalogue.Create(sess.persistRecordLocked(1)))
1562+
1563+ var churn atomic.Bool
1564+ churn.Store(true)
1565+ hookCalls := 0
1566+ d.afterAttachmentEffectParticipantsSnapshotted = func(action string, _ []*attachedClient) {
1567+ if action != "" || !churn.Load() {
1568+ return
1569+ }
1570+ hookCalls++
1571+ if hookCalls%2 == 1 {
1572+ require.True(t, d.detachIfCurrentTransport(sess, ac, ac.transportSnapshot()))
1573+ return
1574+ }
1575+ // Resumed incarnation: re-register the attachment after the first kill
1576+ // snapshotted the detached set.
1577+ d.mu.Lock()
1578+ sess.mu.Lock()
1579+ if sess.attachments == nil {
1580+ sess.attachments = make(map[*attachedClient]struct{})
1581+ }
1582+ sess.attachments[ac] = struct{}{}
1583+ sess.mu.Unlock()
1584+ d.mu.Unlock()
1585+ ac.setSession(sess)
1586+ }
1587+ t.Cleanup(func() { d.afterAttachmentEffectParticipantsSnapshotted = nil })
1588+
1589+ result := d.purgeAllSessions(protocol.ReasonSessionKilled)
1590+ require.True(t, result.Failed(), result.summary())
1591+ require.False(t, result.Superseded())
1592+ require.Len(t, result.Failures, 1)
1593+ require.Equal(t, purgeRecordLive, result.Failures[0].Class)
1594+ require.Equal(t, "work", result.Failures[0].Name, "a churned unit must keep its exact name")
1595+ require.ErrorIs(t, result.Failures[0].Err, errSessionKillParticipantsChanged)
1596+ require.Equal(t, 2, hookCalls, "the churn must outlast the single retry")
1597+ require.Equal(t, 1, sessionCount(d), "a churned unit must stay live")
1598+ d.mu.Lock()
1599+ taken := d.nameLiveOrStoppedLocked("work")
1600+ _, stopped := d.inactive["work"]
1601+ d.mu.Unlock()
1602+ require.True(t, taken, "the churned name must remain taken and accurately reported")
1603+ require.False(t, stopped, "a churned unit must not leave phantom stopped authority")
1604+ requirePurgeAdmissionBalanced(t, d)
1605+ _, ok, err := catalogue.Record("work")
1606+ require.NoError(t, err)
1607+ require.True(t, ok, "a churned unit must keep its durable authority for the retry")
1608+
1609+ churn.Store(false)
1610+ retry := d.purgeAllSessions(protocol.ReasonSessionKilled)
1611+ require.False(t, retry.Failed(), retry.summary())
1612+ require.Equal(t, 0, sessionCount(d), "the churn-free retry must remove the unit")
1613+ _, ok, err = catalogue.Record("work")
1614+ require.NoError(t, err)
1615+ require.False(t, ok, "the churn-free retry must remove durable authority")
1616+ d.sessWg.Wait()
1617+ d.waitNotifies()
1618+}
1619+
1620+// TestPurgeAllAlreadyRemovedParticipantChangeIsSuccess proves the
1621+// already-removed half of the contract: a session a racing lifecycle path
1622+// removed while the purge's participant snapshot went stale owns nothing, so
1623+// the purge reports success rather than a phantom live failure.
1624+func TestPurgeAllAlreadyRemovedParticipantChangeIsSuccess(t *testing.T) {
1625+ d, sess, _, _ := newManualSessionWithPTYs(t, newQuietPTY())
1626+ catalogue := newDurableRecoveryCatalogue(nil)
1627+ WithCatalogue(catalogue, nil)(d)
1628+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, noOpSnapshotRepository{}, nil))(d)
1629+ require.NoError(t, catalogue.Create(sess.persistRecordLocked(1)))
1630+
1631+ d.afterAttachmentEffectParticipantsSnapshotted = func(action string, _ []*attachedClient) {
1632+ if action != "" {
1633+ return
1634+ }
1635+ d.mu.Lock()
1636+ d.unregisterSessionLocked(sess)
1637+ d.mu.Unlock()
1638+ }
1639+ t.Cleanup(func() { d.afterAttachmentEffectParticipantsSnapshotted = nil })
1640+
1641+ result := d.purgeAllSessions(protocol.ReasonSessionKilled)
1642+ require.False(t, result.Failed(), result.summary())
1643+ require.False(t, result.Superseded())
1644+ require.Equal(t, 0, sessionCount(d), "the racing path already removed the session")
1645+ d.mu.Lock()
1646+ taken := d.nameLiveOrStoppedLocked("work")
1647+ _, stopped := d.inactive["work"]
1648+ d.mu.Unlock()
1649+ require.False(t, taken, "an already-removed unit must not stay taken")
1650+ require.False(t, stopped, "an already-removed unit must not leave phantom stopped authority")
1651+ d.sessWg.Wait()
1652+ d.waitNotifies()
1653+}
1654+
1655+// TestPurgeAllStoppedSweepDeadlineIsTypedAndReleasesGate proves the stopped and
1656+// broken sweep spends its own bounded budget: a repository deletion that
1657+// ignores cancellation cannot extend the purge, every record the sweep did not
1658+// finish is reported as its typed stopped/broken failure with its exact name,
1659+// admission is released, the daemon keeps serving, and its next purge removes
1660+// the leftovers.
1661+func TestPurgeAllStoppedSweepDeadlineIsTypedAndReleasesGate(t *testing.T) {
1662+ clock := &signalClock{timers: make(chan *signalTimer, 64)}
1663+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), clock)
1664+ catalogue := newDurableRecoveryCatalogue(nil)
1665+ repository := newBlockingPurgeDeleteRepository()
1666+ WithCatalogue(catalogue, nil)(d)
1667+ WithSnapshotRepository(repository)(d)
1668+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, repository, nil))(d)
1669+ seedStoppedPurgeRecords(t, d, catalogue)
1670+
1671+ purged := make(chan purgeResult, 1)
1672+ go func() { purged <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
1673+
1674+ admission := awaitTestValue(t, clock.timers, "purge did not arm its admission deadline")
1675+ stoppedBudget := awaitTestValue(t, clock.timers, "purge did not arm a separate stopped sweep budget")
1676+ require.NotSame(t, admission, stoppedBudget, "the sweep must spend its own budget, not the admission drain's")
1677+ awaitTestValue(t, repository.entered, "the sweep did not start its uncooperative delete")
1678+ stoppedBudget.ch <- time.Now()
1679+
1680+ result := awaitTestValue(t, purged, "purge did not return after the stopped sweep budget expired")
1681+ require.True(t, result.Failed(), result.summary())
1682+ require.False(t, result.Superseded())
1683+ require.Len(t, result.Failures, 2, "every unfinished stopped/broken record must be reported")
1684+ require.Equal(t, purgeRecordStopped, result.Failures[0].Class)
1685+ require.Equal(t, "stopped", result.Failures[0].Name)
1686+ require.ErrorIs(t, result.Failures[0].Err, errSessionPurgeSweepDeadline)
1687+ require.Equal(t, purgeRecordBroken, result.Failures[1].Class)
1688+ require.Equal(t, "broken", result.Failures[1].Name)
1689+ require.ErrorIs(t, result.Failures[1].Err, errSessionPurgeSweepDeadline)
1690+
1691+ d.mu.Lock()
1692+ admissionStillClosed := d.purgeAdmissionClosing
1693+ daemonClosing := d.closing
1694+ _, stoppedRemains := d.inactive["stopped"]
1695+ _, brokenRemains := d.inactive["broken"]
1696+ d.mu.Unlock()
1697+ require.False(t, admissionStillClosed, "a bounded sweep must reopen purge admission promptly")
1698+ require.False(t, daemonClosing, "a bounded sweep must leave the daemon serving")
1699+ requirePurgeAdmissionBalanced(t, d)
1700+ require.True(t, stoppedRemains, "an unfinished stopped record must keep its authority for the retry")
1701+ require.True(t, brokenRemains, "an unfinished broken record must keep its authority for the retry")
1702+
1703+ close(repository.release)
1704+ // The detached sweep finishes the delete it had already started and then
1705+ // stops; the leftover record is purged by the next bounded pass.
1706+ require.Eventually(t, func() bool {
1707+ d.mu.Lock()
1708+ defer d.mu.Unlock()
1709+ _, broken := d.inactive["broken"]
1710+ return !broken
1711+ }, testWaitTimeout, time.Millisecond, "the detached sweep did not finish the delete it had started")
1712+ require.False(t, d.purgeAllSessions(protocol.ReasonSessionKilled).Failed())
1713+ require.Equal(t, 0, sessionCount(d))
1714+ d.mu.Lock()
1715+ require.Empty(t, d.inactive, "the next purge must remove the leftover records")
1716+ d.mu.Unlock()
1717+ d.sessWg.Wait()
1718+ d.waitNotifies()
1719+}
1720+
1721+// TestPurgeAllStoppedAndLiveBudgetsAreIndependent proves the stopped/broken
1722+// sweep, the live teardown, and the admission drain each spend a distinct
1723+// budget: the stopped budget expiring leaves a parked live unit untouched and
1724+// arms a fresh live budget, whose expiry then reports that live unit.
1725+func TestPurgeAllStoppedAndLiveBudgetsAreIndependent(t *testing.T) {
1726+ clock := &signalClock{timers: make(chan *signalTimer, 64)}
1727+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), clock)
1728+ catalogue := newDurableRecoveryCatalogue(nil)
1729+ repository := newBlockingPurgeDeleteRepository()
1730+ WithCatalogue(catalogue, nil)(d)
1731+ WithSnapshotRepository(repository)(d)
1732+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, repository, nil))(d)
1733+ seedStoppedPurgeRecords(t, d, catalogue)
1734+ live := seedPurgeableLiveSession(t, d, catalogue, "work")
1735+
1736+ purged := make(chan purgeResult, 1)
1737+ go func() { purged <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
1738+
1739+ admission := awaitTestValue(t, clock.timers, "purge did not arm its admission deadline")
1740+ stoppedBudget := awaitTestValue(t, clock.timers, "purge did not arm a stopped sweep budget")
1741+ require.NotSame(t, admission, stoppedBudget)
1742+ awaitTestValue(t, repository.entered, "the sweep did not start its uncooperative delete")
1743+ stoppedBudget.ch <- time.Now()
1744+
1745+ liveBudget := awaitTestValue(t, clock.timers, "the expired sweep did not arm a fresh live budget")
1746+ require.NotSame(t, admission, liveBudget)
1747+ require.NotSame(t, stoppedBudget, liveBudget)
1748+ liveBudget.ch <- time.Now()
1749+
1750+ result := awaitTestValue(t, purged, "purge did not return after both budgets expired")
1751+ require.True(t, result.Failed(), result.summary())
1752+ require.False(t, result.Superseded())
1753+ require.Len(t, result.Failures, 3)
1754+ require.Equal(t, purgeRecordLive, result.Failures[0].Class)
1755+ require.Equal(t, "work", result.Failures[0].Name)
1756+ require.ErrorIs(t, result.Failures[0].Err, errSessionKillDeadline)
1757+ require.Equal(t, purgeRecordStopped, result.Failures[1].Class)
1758+ require.Equal(t, "stopped", result.Failures[1].Name)
1759+ require.ErrorIs(t, result.Failures[1].Err, errSessionPurgeSweepDeadline)
1760+ require.Equal(t, purgeRecordBroken, result.Failures[2].Class)
1761+ require.Equal(t, "broken", result.Failures[2].Name)
1762+ require.ErrorIs(t, result.Failures[2].Err, errSessionPurgeSweepDeadline)
1763+ require.Equal(t, 1, sessionCount(d), "the swept live budget must leave the live unit untouched")
1764+
1765+ close(repository.release)
1766+ live.finishTeardown()
1767+ // The detached sweep finishes the delete it had already started and then
1768+ // stops; the leftover record is purged by the next bounded pass.
1769+ require.Eventually(t, func() bool {
1770+ d.mu.Lock()
1771+ defer d.mu.Unlock()
1772+ _, broken := d.inactive["broken"]
1773+ return !broken
1774+ }, testWaitTimeout, time.Millisecond, "the detached sweep did not finish after release")
1775+ require.False(t, d.purgeAllSessions(protocol.ReasonSessionKilled).Failed())
1776+ require.Equal(t, 0, sessionCount(d))
1777+ d.sessWg.Wait()
1778+ d.waitNotifies()
1779+}
1780+
1781+// TestPurgeAllStoppedSweepDeadlineDefersToExplicitShutdown proves explicit stop
1782+// supersedes the sweep classification: when a shutdown begins as the stopped
1783+// budget expires, the purge reports supersession rather than typed
1784+// stopped/broken failures, so the stop keeps authority over every record.
1785+func TestPurgeAllStoppedSweepDeadlineDefersToExplicitShutdown(t *testing.T) {
1786+ clock := &signalClock{timers: make(chan *signalTimer, 64)}
1787+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), clock)
1788+ catalogue := newDurableRecoveryCatalogue(nil)
1789+ repository := newBlockingPurgeDeleteRepository()
1790+ WithCatalogue(catalogue, nil)(d)
1791+ WithSnapshotRepository(repository)(d)
1792+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, repository, nil))(d)
1793+ seedStoppedPurgeRecords(t, d, catalogue)
1794+
1795+ purged := make(chan purgeResult, 1)
1796+ go func() { purged <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
1797+
1798+ _ = awaitTestValue(t, clock.timers, "purge did not arm its admission deadline")
1799+ stoppedBudget := awaitTestValue(t, clock.timers, "purge did not arm a stopped sweep budget")
1800+ awaitTestValue(t, repository.entered, "the sweep did not start its uncooperative delete")
1801+
1802+ // The stop begins exactly as the sweep budget expires and is deliberately not
1803+ // signalled through the admission change channel, so the deadline is the only
1804+ // select case that can fire.
1805+ d.mu.Lock()
1806+ d.closing = true
1807+ d.mu.Unlock()
1808+ stoppedBudget.ch <- time.Now()
1809+
1810+ result := awaitTestValue(t, purged, "purge did not return after the sweep budget expired")
1811+ require.True(t, result.Superseded(), "an explicit stop concurrent with the sweep deadline must win")
1812+ require.False(t, result.Failed())
1813+
1814+ d.mu.Lock()
1815+ d.closing = false
1816+ stoppedRemains := len(d.inactive)
1817+ d.mu.Unlock()
1818+ require.Equal(t, 2, stoppedRemains, "a superseded sweep must leave every record for the stop")
1819+ close(repository.release)
1820+ d.sessWg.Wait()
1821+ d.waitNotifies()
1822+}
1823+
1824+// TestPurgeAllParticipantChangeDefersToConcurrentExplicitShutdown proves
1825+// explicit stop supersedes the participant-change classification: when a
1826+// shutdown begins as the kill aborts, the purge neither retries against the
1827+// stop nor reports a live failure, so the stop preserves the session.
1828+func TestPurgeAllParticipantChangeDefersToConcurrentExplicitShutdown(t *testing.T) {
1829+ d, sess, ac, _ := newManualSessionWithPTYs(t, newQuietPTY())
1830+ catalogue := newDurableRecoveryCatalogue(nil)
1831+ WithCatalogue(catalogue, nil)(d)
1832+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, noOpSnapshotRepository{}, nil))(d)
1833+ require.NoError(t, catalogue.Create(sess.persistRecordLocked(1)))
1834+
1835+ // The racing path changes the participants and begins an explicit stop at
1836+ // once, exactly as the kill's post-freeze revalidation runs.
1837+ hookCalls := 0
1838+ d.afterAttachmentEffectParticipantsSnapshotted = func(action string, _ []*attachedClient) {
1839+ if action != "" {
1840+ return
1841+ }
1842+ hookCalls++
1843+ require.True(t, d.detachIfCurrentTransport(sess, ac, ac.transportSnapshot()))
1844+ d.mu.Lock()
1845+ d.closing = true
1846+ d.mu.Unlock()
1847+ }
1848+ t.Cleanup(func() { d.afterAttachmentEffectParticipantsSnapshotted = nil })
1849+
1850+ result := d.purgeAllSessions(protocol.ReasonSessionKilled)
1851+ require.True(t, result.Superseded(), "an explicit stop concurrent with the participant change must win")
1852+ require.False(t, result.Failed())
1853+ require.Equal(t, 1, hookCalls, "a superseded purge must not retry against the stop")
1854+ require.Equal(t, 1, sessionCount(d), "a superseded purge must leave the session for the stop")
1855+
1856+ d.mu.Lock()
1857+ d.closing = false
1858+ _, stopped := d.inactive["work"]
1859+ d.mu.Unlock()
1860+ require.False(t, stopped, "a superseded purge must not leave phantom stopped authority")
1861+ d.sessWg.Wait()
1862+ d.waitNotifies()
1863+}
1864+
1865+// TestPurgeAllStopBetweenPrecheckAndOwnershipIsSupersededNotSuccess proves the
1866+// no-false-success contract at the last window: when an explicit stop wins a
1867+// live unit after the purge's per-unit precheck but before that unit attempts
1868+// teardown ownership, purgeLiveSession sees an already-removed target and is a
1869+// no-op. The purge must defer to the stop and report supersession instead of
1870+// counting the stop-owned unit as a successful purge.
1871+func TestPurgeAllStopBetweenPrecheckAndOwnershipIsSupersededNotSuccess(t *testing.T) {
1872+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), stubClock{})
1873+ catalogue := newDurableRecoveryCatalogue(nil)
1874+ WithCatalogue(catalogue, nil)(d)
1875+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, noOpSnapshotRepository{}, nil))(d)
1876+
1877+ live := newSnapshotTestSession(t, "work", false, "/work")
1878+ record := live.persistRecordLocked(live.createdAt)
1879+ require.NoError(t, catalogue.Create(record))
1880+ livePTY, ok := live.tabs[0].panes["pane-1"].pty.(*portsmocks.MockPTY)
1881+ require.True(t, ok)
1882+ livePTY.EXPECT().Close().Return(nil).Maybe()
1883+ d.sessions[live.id] = live
1884+
1885+ d.beforePurgeLiveSessionKill = func(sess *session) {
1886+ d.beforePurgeLiveSessionKill = nil
1887+ // The explicit stop wins the exact unit in this window: it marks the
1888+ // daemon closing, removes the session from the live registry, and
1889+ // preserves its stopped authority, exactly as its own teardown does.
1890+ d.mu.Lock()
1891+ d.closing = true
1892+ d.signalPurgeAdmissionChangedLocked()
1893+ d.unregisterSessionLocked(sess)
1894+ d.inactive["work"] = inactiveSessionFromRecord(record, protocol.SessionDown, nil)
1895+ d.mu.Unlock()
1896+ }
1897+ t.Cleanup(func() { d.beforePurgeLiveSessionKill = nil })
1898+
1899+ result := d.purgeAllSessions(protocol.ReasonSessionKilled)
1900+ require.True(t, result.Superseded(), "a stop that wins the unit before ownership must supersede the purge")
1901+ require.False(t, result.Failed(), result.summary())
1902+ require.Equal(t, 0, sessionCount(d), "the stop owns the session from this point")
1903+ requirePurgeAdmissionBalanced(t, d)
1904+
1905+ d.mu.Lock()
1906+ _, stopped := d.inactive["work"]
1907+ closing := d.closing
1908+ d.mu.Unlock()
1909+ require.True(t, stopped, "the stop's preserved authority must survive the purge")
1910+ require.True(t, closing, "the stop must stay in effect")
1911+ _, ok, err := catalogue.Record("work")
1912+ require.NoError(t, err)
1913+ require.True(t, ok, "a superseded purge must not delete durable authority")
1914+ d.waitNotifies()
1915+}
1916+
1917+// blockingPublicationRepository blocks its first publication until released and
1918+// deliberately ignores context cancellation, modelling an uncooperative
1919+// publisher whose repository call outlives the shared teardown budget.
1920+type blockingPublicationRepository struct {
1921+ noOpSnapshotRepository
1922+
1923+ entered chan struct{}
1924+ release chan struct{}
1925+ once sync.Once
1926+}
1927+
1928+func newBlockingPublicationRepository() *blockingPublicationRepository {
1929+ return &blockingPublicationRepository{entered: make(chan struct{}), release: make(chan struct{})}
1930+}
1931+
1932+func (r *blockingPublicationRepository) Publish(context.Context, ports.SnapshotPublication) error {
1933+ r.once.Do(func() { close(r.entered) })
1934+ <-r.release
1935+ return nil
1936+}
1937+
1938+// TestPurgeAllLiveQuarantineJoinBoundedByDeadlineReleasesAdmission proves the
1939+// purge's snapshot-coordinator quarantine join spends the shared live budget:
1940+// an uncooperative publisher that ignores cancellation can never hold purge
1941+// admission open, the live unit is reported as a typed skip instead of being
1942+// counted as purged, and a later purge removes it once the publisher returns.
1943+func TestPurgeAllLiveQuarantineJoinBoundedByDeadlineReleasesAdmission(t *testing.T) {
1944+ clock := &signalClock{timers: make(chan *signalTimer, 64)}
1945+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), clock)
1946+ catalogue := newDurableRecoveryCatalogue(nil)
1947+ repository := newBlockingPublicationRepository()
1948+ WithCatalogue(catalogue, nil)(d)
1949+ WithSnapshotRepository(repository)(d)
1950+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, repository, nil))(d)
1951+ startSnapshotEncodeWorker(t, d)
1952+
1953+ var releaseOnce sync.Once
1954+ release := func() { releaseOnce.Do(func() { close(repository.release) }) }
1955+ // Register after the worker cleanup so the uncooperative publication is
1956+ // always released before cleanup asks the worker to join.
1957+ t.Cleanup(release)
1958+
1959+ live := newSnapshotTestSession(t, "work", false, "/work")
1960+ require.NoError(t, catalogue.Create(live.persistRecordLocked(live.createdAt)))
1961+ livePTY, ok := live.tabs[0].panes["pane-1"].pty.(*portsmocks.MockPTY)
1962+ require.True(t, ok)
1963+ livePTY.EXPECT().Close().Return(nil).Maybe()
1964+ d.sessions[live.id] = live
1965+
1966+ markSnapshotDirty(live)
1967+ require.True(t, d.scheduleSnapshot(live))
1968+ awaitTestValue(t, repository.entered, "the live publication did not start")
1969+
1970+ purged := make(chan purgeResult, 1)
1971+ go func() { purged <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
1972+
1973+ _ = awaitTestValue(t, clock.timers, "purge did not arm its admission deadline")
1974+ liveBudget := awaitTestValue(t, clock.timers, "purge did not arm a separate live-teardown budget")
1975+ liveBudget.ch <- time.Now()
1976+
1977+ result := awaitTestValue(t, purged, "purge did not return while the publisher stayed blocked")
1978+ require.True(t, result.Failed(), result.summary())
1979+ require.False(t, result.Superseded())
1980+ require.Len(t, result.Failures, 1)
1981+ require.Equal(t, purgeRecordLive, result.Failures[0].Class)
1982+ require.Equal(t, "work", result.Failures[0].Name)
1983+ require.ErrorIs(t, result.Failures[0].Err, errSessionKillDeadline)
1984+ require.Equal(t, 1, sessionCount(d), "a skipped live unit must stay live")
1985+ requirePurgeAdmissionBalanced(t, d)
1986+
1987+ // The retry succeeds once the uncooperative publisher returns: the bound
1988+ // released admission without discarding ownership of the unit.
1989+ release()
1990+ awaitSnapshotIdle(t, live)
1991+ require.False(t, d.purgeAllSessions(protocol.ReasonSessionKilled).Failed())
1992+ require.Equal(t, 0, sessionCount(d), "the retried purge must remove the unit")
1993+ d.mu.Lock()
1994+ _, stopped := d.inactive["work"]
1995+ d.mu.Unlock()
1996+ require.False(t, stopped, "the retried purge must not leave phantom stopped authority")
1997+ d.sessWg.Wait()
1998+ d.waitNotifies()
1999+}
2000+
2001+// TestPurgeAllLiveQuarantineAbortRestoresSurvivingSession proves the reversible
2002+// half of the quarantine contract: when the shared live budget expires while a
2003+// purge is joining an uncooperative snapshot publication, the abort rolls the
2004+// coordinator quarantine back instead of leaving a registered live session
2005+// permanently ineligible. The surviving session checkpoints again, an EOF
2006+// teardown completes, and its attached client is never left frozen.
2007+func TestPurgeAllLiveQuarantineAbortRestoresSurvivingSession(t *testing.T) {
2008+ clock := &signalClock{timers: make(chan *signalTimer, 64)}
2009+ d, live, ac, _ := newManualSessionWithPTYsClock(t, clock, newQuietPTY())
2010+ live.createdAt = 42
2011+ live.snapEligible.Store(true)
2012+
2013+ catalogue := newDurableRecoveryCatalogue(nil)
2014+ repository := newBlockingPublicationRepository()
2015+ WithCatalogue(catalogue, nil)(d)
2016+ WithSnapshotRepository(repository)(d)
2017+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, repository, nil))(d)
2018+ startSnapshotEncodeWorker(t, d)
2019+
2020+ var releaseOnce sync.Once
2021+ release := func() { releaseOnce.Do(func() { close(repository.release) }) }
2022+ // Register after the worker cleanup so the uncooperative publication is
2023+ // always released before cleanup asks the worker to join.
2024+ t.Cleanup(release)
2025+
2026+ require.NoError(t, catalogue.Create(live.persistRecordLocked(live.createdAt)))
2027+
2028+ // Park an uncooperative publication so the purge's coordinator join cannot
2029+ // finish inside the shared live budget.
2030+ markSnapshotDirty(live)
2031+ require.True(t, d.scheduleSnapshot(live))
2032+ awaitTestValue(t, repository.entered, "the live publication did not start")
2033+
2034+ purged := make(chan purgeResult, 1)
2035+ go func() { purged <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
2036+
2037+ _ = awaitTestValue(t, clock.timers, "purge did not arm its admission deadline")
2038+ liveBudget := awaitTestValue(t, clock.timers, "purge did not arm a separate live-teardown budget")
2039+ liveBudget.ch <- time.Now()
2040+
2041+ result := awaitTestValue(t, purged, "purge did not return while the publisher stayed blocked")
2042+ require.True(t, result.Failed(), result.summary())
2043+ require.False(t, result.Superseded())
2044+ require.Len(t, result.Failures, 1)
2045+ require.Equal(t, purgeRecordLive, result.Failures[0].Class)
2046+ require.Equal(t, "work", result.Failures[0].Name)
2047+ require.ErrorIs(t, result.Failures[0].Err, errSessionKillDeadline)
2048+ require.Equal(t, 1, sessionCount(d), "a skipped live unit must stay live")
2049+ requirePurgeAdmissionBalanced(t, d)
2050+
2051+ // The abort rolled the quarantine back: the surviving session is eligible
2052+ // and dirty again, and its attached client's gate is stable, not frozen.
2053+ require.True(t, live.snapEligible.Load(), "an aborted purge must restore snapshot eligibility")
2054+ live.snapshotMu.Lock()
2055+ quarantined := live.snapshotQuarantined
2056+ live.snapshotMu.Unlock()
2057+ require.False(t, quarantined, "an aborted purge must roll back the coordinator quarantine")
2058+ require.True(t, live.snapDirty.Load(), "an aborted purge must keep the surviving session dirty")
2059+ ac.lifecycle.mu.Lock()
2060+ phase := ac.lifecycle.phase
2061+ ac.lifecycle.mu.Unlock()
2062+ require.Equal(t, attachmentEffectsStable, phase, "an aborted purge must not leave the attached client frozen")
2063+
2064+ // Once the uncooperative publication returns, the surviving session must
2065+ // checkpoint again.
2066+ release()
2067+ awaitSnapshotIdle(t, live)
2068+ // The fake clock's routine pacing interval is zero-now based, so clear it and
2069+ // prove the rolled-back coordinator schedules a real replacement capture.
2070+ live.snapshotMu.Lock()
2071+ live.snapshotNextEligibleAt = time.Time{}
2072+ live.snapshotMu.Unlock()
2073+ require.True(t, d.scheduleSnapshot(live), "the surviving session must schedule a replacement checkpoint")
2074+ awaitSnapshotClean(t, live)
2075+ _, ok, err := catalogue.Record("work")
2076+ require.NoError(t, err)
2077+ require.True(t, ok, "the surviving session must keep its durable authority")
2078+
2079+ // An EOF teardown of the surviving session must complete.
2080+ require.NoError(t, d.killSession(live, protocol.ReasonSessionKilled, false))
2081+ require.Equal(t, 0, sessionCount(d), "the EOF teardown must remove the surviving session")
2082+ d.sessWg.Wait()
2083+ d.waitNotifies()
2084+}
2085+
2086+// TestSessionKillPostQuarantineParticipantMismatchRestoresSurvivingSession is
2087+// the deterministic post-quarantine half of the reversible teardown contract:
2088+// once a direct session kill or last-tab close has quarantined the snapshot
2089+// coordinator, a participant change caught by the pre-publication revalidation
2090+// aborts without removing anything and rolls the quarantine back. The surviving
2091+// session resumes checkpoint scheduling and a later teardown completes.
2092+func TestSessionKillPostQuarantineParticipantMismatchRestoresSurvivingSession(t *testing.T) {
2093+ tests := []struct {
2094+ name string
2095+ kill func(t *testing.T, d *Daemon, sess *session) error
2096+ }{
2097+ {
2098+ name: "direct kill",
2099+ kill: func(_ *testing.T, d *Daemon, sess *session) error {
2100+ return d.killSession(sess, protocol.ReasonSessionKilled, true)
2101+ },
2102+ },
2103+ {
2104+ name: "close tab",
2105+ kill: func(_ *testing.T, d *Daemon, sess *session) error {
2106+ return d.closeTab(sess, sess.tabs[0], false)
2107+ },
2108+ },
2109+ }
2110+ for _, tt := range tests {
2111+ t.Run(tt.name, func(t *testing.T) {
2112+ d, sess, ac, _ := newManualSessionWithPTYs(t, newQuietPTY())
2113+ sess.snapEligible.Store(true)
2114+ catalogue := newDurableRecoveryCatalogue(nil)
2115+ repository := noOpSnapshotRepository{}
2116+ WithCatalogue(catalogue, nil)(d)
2117+ WithSnapshotRepository(repository)(d)
2118+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, repository, nil))(d)
2119+ startSnapshotEncodeWorker(t, d)
2120+ require.NoError(t, catalogue.Create(sess.persistRecordLocked(sess.createdAt)))
2121+
2122+ d.afterSessionKillQuarantine = func(*session) {
2123+ d.afterSessionKillQuarantine = nil
2124+ // A token resume replaces the exact transport after the snapshot
2125+ // coordinator was quarantined but before its publication revalidation,
2126+ // so the teardown must abort and roll the quarantine back. The
2127+ // attachment gate is frozen here, so only the transport incarnation
2128+ // changes; the capability is rebound by the next real claim.
2129+ fresh := &closeTrackingTransport{}
2130+ ac.replaceTransport(fresh)
2131+ }
2132+ t.Cleanup(func() { d.afterSessionKillQuarantine = nil })
2133+
2134+ require.ErrorIs(t, tt.kill(t, d, sess), errSessionKillParticipantsChanged)
2135+ require.Equal(t, 1, sessionCount(d), "an aborted teardown must leave the session registered")
2136+
2137+ require.True(t, sess.snapEligible.Load(), "the abort must restore snapshot eligibility")
2138+ sess.snapshotMu.Lock()
2139+ quarantined := sess.snapshotQuarantined
2140+ sess.snapshotMu.Unlock()
2141+ require.False(t, quarantined, "the abort must roll back the coordinator quarantine")
2142+ require.True(t, sess.snapDirty.Load(), "the abort must keep the surviving session checkpointable")
2143+
2144+ // The surviving session checkpoints again.
2145+ require.True(t, d.scheduleSnapshot(sess), "the surviving session must schedule a checkpoint")
2146+ awaitSnapshotClean(t, sess)
2147+ _, ok, err := catalogue.Record("work")
2148+ require.NoError(t, err)
2149+ require.True(t, ok, "the surviving session must keep its durable authority")
2150+
2151+ // A later teardown of the surviving session completes.
2152+ require.NoError(t, d.killSession(sess, protocol.ReasonSessionKilled, true))
2153+ require.Equal(t, 0, sessionCount(d), "the later teardown must remove the surviving session")
2154+ d.sessWg.Wait()
2155+ d.waitNotifies()
2156+ })
2157+ }
2158+}
2159+
2160+// blockingLiveDeleteRepository blocks its incarnation delete until released,
2161+// ignores context cancellation, and signals when the blocked call returns, so a
2162+// test can observe a detached live exact delete outliving the bounded purge.
2163+type blockingLiveDeleteRepository struct {
2164+ noOpSnapshotRepository
2165+
2166+ entered chan struct{}
2167+ release chan struct{}
2168+ returned chan struct{}
2169+ // err is returned once release closes, letting a test model a detached
2170+ // delete that fails only after the bounded purge already gave up waiting.
2171+ err error
2172+ once sync.Once
2173+}
2174+
2175+func newBlockingLiveDeleteRepository() *blockingLiveDeleteRepository {
2176+ return &blockingLiveDeleteRepository{
2177+ entered: make(chan struct{}),
2178+ release: make(chan struct{}),
2179+ returned: make(chan struct{}),
2180+ }
2181+}
2182+
2183+func (r *blockingLiveDeleteRepository) DeleteIncarnation(context.Context, domain.IncarnationID) error {
2184+ r.once.Do(func() { close(r.entered) })
2185+ <-r.release
2186+ select {
2187+ case <-r.returned:
2188+ default:
2189+ close(r.returned)
2190+ }
2191+ return r.err
2192+}
2193+
2194+// TestPurgeAllLiveDurableDeleteBoundedDetachesAndReleasesGate proves a live
2195+// unit's uncooperative exact durable delete cannot hold purgeAllMu, admission,
2196+// or control: the purge returns a typed pending failure at the shared live
2197+// budget while the live identity is already removed and its stopped authority
2198+// still reserves the name. A second purge is not queued behind the blocked
2199+// delete, and once the stale delete returns the name is free for a later
2200+// same-name lifecycle.
2201+func TestPurgeAllLiveDurableDeleteBoundedDetachesAndReleasesGate(t *testing.T) {
2202+ clock := &signalClock{timers: make(chan *signalTimer, 64)}
2203+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), clock)
2204+ catalogue := newDurableRecoveryCatalogue(nil)
2205+ repository := newBlockingLiveDeleteRepository()
2206+ WithCatalogue(catalogue, nil)(d)
2207+ WithSnapshotRepository(repository)(d)
2208+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, repository, nil))(d)
2209+
2210+ live := newSnapshotTestSession(t, "work", false, "/work")
2211+ require.NoError(t, catalogue.Create(live.persistRecordLocked(live.createdAt)))
2212+ livePTY, ok := live.tabs[0].panes["pane-1"].pty.(*portsmocks.MockPTY)
2213+ require.True(t, ok)
2214+ livePTY.EXPECT().Close().Return(nil).Maybe()
2215+ d.sessions[live.id] = live
2216+
2217+ purged := make(chan purgeResult, 1)
2218+ go func() { purged <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
2219+
2220+ _ = awaitTestValue(t, clock.timers, "purge did not arm its admission deadline")
2221+ liveBudget := awaitTestValue(t, clock.timers, "purge did not arm a separate live-teardown budget")
2222+ awaitTestValue(t, repository.entered, "the live purge did not start its uncooperative delete")
2223+ liveBudget.ch <- time.Now()
2224+
2225+ result := awaitTestValue(t, purged, "purge did not return while the live durable delete stayed blocked")
2226+ require.True(t, result.Failed(), result.summary())
2227+ require.False(t, result.Superseded())
2228+ require.Len(t, result.Failures, 1)
2229+ require.Equal(t, purgeRecordLive, result.Failures[0].Class)
2230+ require.Equal(t, "work", result.Failures[0].Name, "the pending delete must keep its exact name")
2231+ require.ErrorIs(t, result.Failures[0].Err, errSessionPurgeDeletePending)
2232+
2233+ require.Equal(t, 0, sessionCount(d), "the live identity must be removed before its durable delete")
2234+ requirePurgeAdmissionBalanced(t, d)
2235+ d.mu.Lock()
2236+ closing := d.closing
2237+ _, stoppedRemains := d.inactive["work"]
2238+ d.mu.Unlock()
2239+ require.False(t, closing, "a bounded live delete must leave the daemon serving")
2240+ require.True(t, stoppedRemains, "a pending exact delete must reserve the name with stopped authority")
2241+
2242+ // A second purge is not queued behind the blocked delete: it takes the purge
2243+ // admission gate and runs while the stale delete still holds the coordinator
2244+ // lock, so it is bounded by its own sweep instead of waiting on purgeAllMu.
2245+ second := make(chan purgeResult, 1)
2246+ go func() { second <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
2247+ awaitPurgeGateClosed(t, d)
2248+
2249+ // Releasing the stale delete lets the detached first finish and the second
2250+ // purge settle. The durable record was already gone, so the second purge
2251+ // clears the stale stopped authority without a new repository delete.
2252+ close(repository.release)
2253+ awaitTestValue(t, repository.returned, "the detached exact delete did not return")
2254+ secondResult := awaitTestValue(t, second, "the second purge did not complete after the stale delete returned")
2255+ require.False(t, secondResult.Failed(), secondResult.summary())
2256+ require.False(t, secondResult.Superseded())
2257+ d.mu.Lock()
2258+ _, stoppedRemains = d.inactive["work"]
2259+ taken := d.nameLiveOrStoppedLocked("work")
2260+ d.mu.Unlock()
2261+ require.False(t, stoppedRemains, "the second purge must clear the stale stopped authority")
2262+ require.False(t, taken, "the purged name must be free for a later same-name lifecycle")
2263+
2264+ // A later same-name lifecycle is safe to adopt and keeps its own authority.
2265+ replacement := newSnapshotTestSession(t, "work", false, "/work")
2266+ replacement.incarnation = domain.IncarnationID{2}
2267+ replacement.createdAt = 43
2268+ require.NoError(t, catalogue.Create(replacement.persistRecordLocked(replacement.createdAt)))
2269+ replacementPTY, ok := replacement.tabs[0].panes["pane-1"].pty.(*portsmocks.MockPTY)
2270+ require.True(t, ok)
2271+ replacementPTY.EXPECT().Close().Return(nil).Maybe()
2272+ d.sessions[replacement.id] = replacement
2273+ record, ok, err := catalogue.Record("work")
2274+ require.NoError(t, err)
2275+ require.True(t, ok, "the replacement must own the same-name durable record")
2276+ require.Equal(t, replacement.incarnation, record.IncarnationID)
2277+ require.Equal(t, 1, sessionCount(d))
2278+ d.sessWg.Wait()
2279+ d.waitNotifies()
2280+}
2281+
2282+// TestPurgeAllDetachedLiveDeleteFencesSameNameReplacement proves the detached
2283+// live exact delete removes only the stopped authority whose name, incarnation,
2284+// and createdAt it captured: a same-name replacement lifecycle that holds the
2285+// name when the stale delete returns is never removed.
2286+func TestPurgeAllDetachedLiveDeleteFencesSameNameReplacement(t *testing.T) {
2287+ clock := &signalClock{timers: make(chan *signalTimer, 64)}
2288+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), clock)
2289+ catalogue := newDurableRecoveryCatalogue(nil)
2290+ repository := newBlockingLiveDeleteRepository()
2291+ WithCatalogue(catalogue, nil)(d)
2292+ WithSnapshotRepository(repository)(d)
2293+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, repository, nil))(d)
2294+
2295+ live := newSnapshotTestSession(t, "work", false, "/work")
2296+ require.NoError(t, catalogue.Create(live.persistRecordLocked(live.createdAt)))
2297+ livePTY, ok := live.tabs[0].panes["pane-1"].pty.(*portsmocks.MockPTY)
2298+ require.True(t, ok)
2299+ livePTY.EXPECT().Close().Return(nil).Maybe()
2300+ d.sessions[live.id] = live
2301+
2302+ purged := make(chan purgeResult, 1)
2303+ go func() { purged <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
2304+
2305+ _ = awaitTestValue(t, clock.timers, "purge did not arm its admission deadline")
2306+ liveBudget := awaitTestValue(t, clock.timers, "purge did not arm a separate live-teardown budget")
2307+ awaitTestValue(t, repository.entered, "the live purge did not start its uncooperative delete")
2308+ liveBudget.ch <- time.Now()
2309+ result := awaitTestValue(t, purged, "purge did not return while the live durable delete stayed blocked")
2310+ require.True(t, result.Failed(), result.summary())
2311+ require.Len(t, result.Failures, 1)
2312+ require.ErrorIs(t, result.Failures[0].Err, errSessionPurgeDeletePending)
2313+
2314+ // A same-name replacement lifecycle takes the reserved name while the stale
2315+ // delete is still blocked.
2316+ replacement := inactiveSession{name: "work", createdAt: 99, incarnation: domain.IncarnationID{9}, purging: true}
2317+ d.mu.Lock()
2318+ d.inactive["work"] = replacement
2319+ d.mu.Unlock()
2320+
2321+ close(repository.release)
2322+ awaitTestValue(t, repository.returned, "the detached exact delete did not return")
2323+ d.mu.Lock()
2324+ current, ok := d.inactive["work"]
2325+ d.mu.Unlock()
2326+ require.True(t, ok, "the detached stale delete must not remove a same-name replacement lifecycle")
2327+ require.Equal(t, replacement.incarnation, current.incarnation)
2328+ d.sessWg.Wait()
2329+ d.waitNotifies()
2330+}
2331+
2332+// TestPurgeAllLogsDetachedLiveDeleteLateFailure proves a detached live exact
2333+// delete that fails only after the bounded purge already returned a typed
2334+// pending verdict is still recorded, using bounded non-sensitive fields instead
2335+// of the user-chosen session name.
2336+func TestPurgeAllLogsDetachedLiveDeleteLateFailure(t *testing.T) {
2337+ clock := &signalClock{timers: make(chan *signalTimer, 64)}
2338+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), clock)
2339+ var logged syncedLogBuffer
2340+ d.log = slog.New(slog.NewTextHandler(&logged, nil))
2341+ catalogue := newDurableRecoveryCatalogue(nil)
2342+ repository := newBlockingLiveDeleteRepository()
2343+ repository.err = errors.New("detached delete failed")
2344+ WithCatalogue(catalogue, nil)(d)
2345+ WithSnapshotRepository(repository)(d)
2346+ WithRecoveryCoordinator(recoveryusecase.NewCoordinator(catalogue, repository, nil))(d)
2347+
2348+ live := newSnapshotTestSession(t, "work", false, "/work")
2349+ require.NoError(t, catalogue.Create(live.persistRecordLocked(live.createdAt)))
2350+ livePTY, ok := live.tabs[0].panes["pane-1"].pty.(*portsmocks.MockPTY)
2351+ require.True(t, ok)
2352+ livePTY.EXPECT().Close().Return(nil).Maybe()
2353+ d.sessions[live.id] = live
2354+
2355+ purged := make(chan purgeResult, 1)
2356+ go func() { purged <- d.purgeAllSessions(protocol.ReasonSessionKilled) }()
2357+
2358+ _ = awaitTestValue(t, clock.timers, "purge did not arm its admission deadline")
2359+ liveBudget := awaitTestValue(t, clock.timers, "purge did not arm a separate live-teardown budget")
2360+ awaitTestValue(t, repository.entered, "the live purge did not start its uncooperative delete")
2361+ liveBudget.ch <- time.Now()
2362+
2363+ result := awaitTestValue(t, purged, "purge did not return while the live durable delete stayed blocked")
2364+ require.True(t, result.Failed(), result.summary())
2365+ require.Len(t, result.Failures, 1)
2366+ require.ErrorIs(t, result.Failures[0].Err, errSessionPurgeDeletePending)
2367+
2368+ // The stale delete fails after the bounded caller already returned, and the
2369+ // detached waiter records the late failure.
2370+ close(repository.release)
2371+ awaitTestValue(t, repository.returned, "the detached exact delete did not return")
2372+ require.Eventually(t, func() bool {
2373+ return strings.Contains(logged.String(), "detached session purge delete failed after deadline")
2374+ }, testWaitTimeout, time.Millisecond, "a late detached delete failure must be logged")
2375+
2376+ lateLine := ""
2377+ for _, line := range strings.Split(logged.String(), "\n") {
2378+ if strings.Contains(line, "detached session purge delete failed after deadline") {
2379+ lateLine = line
2380+ }
2381+ }
2382+ require.Contains(t, lateLine, "detached delete failed")
2383+ require.NotContains(t, lateLine, "work", "the late failure must not log the user-chosen session name")
2384+ d.sessWg.Wait()
2385+ d.waitNotifies()
2386+}
M internal/usecase/daemon/remote_attach.go +22 -3
@@ -40,6 +40,10 @@ func (d *Daemon) routeRemoteTargetWithContext(ctx context.Context, h protocol.He
4040 d.mu.Unlock()
4141 return nil, nil, &protoErr{protocol.ErrServerShutdown, "daemon is shutting down"}
4242 }
43+ // Purge admission is already reserved by routeWithContext before this
44+ // dispatch, so an admitted remote target must not re-check the transient
45+ // gate: a purge that begins while this route is admitted waits for it and
46+ // must not reject it.
4347 if live := d.findByNameLocked(target.SessionName); live != nil {
4448 if live.incarnation != target.LifecycleID {
4549 d.mu.Unlock()
@@ -127,12 +131,15 @@ func (d *Daemon) sendCommittedRouteIdentityForAttachment(effect *attachmentEffec
127131 }
128132
129133 func (d *Daemon) finishRouteAttach(sess *session, tr ports.ServerConnection, sz domain.Size, h protocol.Hello, routeCreated, purge bool) (*attachedClient, error) {
130- ac, err := d.finishAttach(sess, tr, sz, h)
131- if err != nil && routeCreated {
134+ ac, attachErr := d.finishAttach(sess, tr, sz, h)
135+ err := attachErr
136+ if attachErr != nil && routeCreated {
132137 if cleanupErr := d.killSessionIfEmpty(sess, protocol.ReasonSessionKilled, purge); cleanupErr != nil {
133138 err = errors.Join(err, cleanupErr)
134139 }
135- return nil, err
140+ }
141+ if verdict, ok := d.routeAttachStopVerdict(attachErr); ok {
142+ return nil, verdict
136143 }
137144 if err == nil && routeCreated && ac != nil {
138145 ac.routeCreatedSession = true
@@ -141,6 +148,18 @@ func (d *Daemon) finishRouteAttach(sess *session, tr ports.ServerConnection, sz
141148 return ac, err
142149 }
143150
151+// routeAttachStopVerdict reports the authoritative explicit-stop outcome when an
152+// attach failure is the leaked transition of a handshake the stop tore down. It
153+// deliberately inspects only attachErr: the rollback cleanup error the caller
154+// may join onto it is internal bookkeeping and must never decide the verdict, so
155+// an unrelated attach failure is reported as-is even while the daemon stops.
156+func (d *Daemon) routeAttachStopVerdict(attachErr error) (error, bool) {
157+ if attachErr == nil || !errors.Is(attachErr, errAttachmentTransition) || !d.purgeSupersededForShutdown() {
158+ return nil, false
159+ }
160+ return &protoErr{protocol.ErrServerShutdown, "daemon is shutting down"}, true
161+}
162+
144163 // remoteTargetMatchesSessionLocked validates the exact lifecycle and tab
145164 // selector while the daemon registry lock is held. Callers use this before
146165 // changing resume ownership so a stale target cannot partially claim a session.
M internal/usecase/daemon/remote_monitor_lifecycle_test.go +29 -9
@@ -156,32 +156,52 @@ func TestServeStartsMonitorWithoutWaitingForRemote(t *testing.T) {
156156 }
157157 }
158158
159-// TestServeIdleExitWithMonitorRunning proves monitoring never counts as an
160-// attachment: the last session exit still drives Serve to return, and the
161-// monitor stops with the daemon.
162-func TestServeIdleExitWithMonitorRunning(t *testing.T) {
159+// TestServeSurvivesIdleExitWithMonitorRunning proves monitoring never counts as
160+// an attachment and the last session exit no longer drives Serve to return: the
161+// daemon keeps serving and the monitor keeps running across the empty
162+// registry, and only explicit cancellation stops both.
163+func TestServeSurvivesIdleExitWithMonitorRunning(t *testing.T) {
163164 monitor := &scriptRemoteMonitor{started: make(chan struct{}), stopped: make(chan struct{})}
164165 d, l, _, sends, releasePTY := newMonitorServeHarness(t, monitor)
166+ defer releasePTY()
165167
168+ ctx, cancel := context.WithCancel(context.Background())
169+ defer cancel()
166170 served := make(chan error, 1)
167- go func() { served <- d.Serve(context.Background(), l) }()
171+ go func() { served <- d.Serve(ctx, l) }()
168172
169173 awaitFrame(t, sends, "Welcome")
170174 awaitFrame(t, sends, "Output")
171175
172- // Child exits -> session removed -> registry empties -> Serve returns,
173- // even though the monitor runner is still parked in its context.
176+ // Child exits -> session removed -> registry empties. Serve must keep
177+ // running and the monitor must stay parked in its context.
174178 releasePTY()
179+ det := awaitFrame(t, sends, "Detached")
180+ require.Equal(t, protocol.ReasonSessionKilled, decodeServerMessage(t, det).(protocol.Detached).Reason)
181+ require.Equal(t, 0, sessionCount(d))
182+ select {
183+ case <-served:
184+ t.Fatal("Serve returned after the last session exited")
185+ default:
186+ }
187+ select {
188+ case <-monitor.stopped:
189+ t.Fatal("monitor stopped while the daemon was still serving")
190+ default:
191+ }
192+
193+ // Only explicit cancellation ends the daemon.
194+ cancel()
175195 select {
176196 case err := <-served:
177197 require.NoError(t, err)
178198 case <-time.After(10 * time.Second):
179- t.Fatal("Serve did not return after last session exited")
199+ t.Fatal("Serve did not return after context cancel")
180200 }
181201 select {
182202 case <-monitor.stopped:
183203 case <-time.After(5 * time.Second):
184- t.Fatal("monitor runner did not stop after idle exit")
204+ t.Fatal("monitor runner did not stop after explicit shutdown")
185205 }
186206 }
187207
M internal/usecase/daemon/remote_route_test.go +119 -0
@@ -2,6 +2,8 @@ package daemon
22
33 import (
44 "context"
5+ "errors"
6+ "fmt"
57 "os"
68 "testing"
79
@@ -222,3 +224,120 @@ func TestRouteRemoteTargetRejectsSameNameReplacement(t *testing.T) {
222224 require.ErrorAs(t, err, &protocolErr)
223225 require.Equal(t, protocol.ErrNoSuchTarget, protocolErr.code)
224226 }
227+
228+// TestRouteAttachStopVerdictKeysOnAttachError proves the supersession rewrite is
229+// keyed on the attach failure: a cleanup error that happens to chain
230+// errAttachmentTransition must not turn an unrelated attach failure into the
231+// shutdown verdict, while a genuine attach-level transition under a stop still
232+// is.
233+func TestRouteAttachStopVerdictKeysOnAttachError(t *testing.T) {
234+ d := newTestDaemon(t, nil, stubClock{})
235+ d.mu.Lock()
236+ d.closing = true
237+ d.mu.Unlock()
238+ t.Cleanup(func() {
239+ d.mu.Lock()
240+ d.closing = false
241+ d.mu.Unlock()
242+ })
243+
244+ attachErr := errors.New("route attach failed")
245+ cleanupErr := fmt.Errorf("route cleanup: %w", errAttachmentTransition)
246+ joined := errors.Join(attachErr, cleanupErr)
247+ // The joined rollback error really does carry the transition sentinel, which
248+ // is exactly what the old rewrite keyed on.
249+ require.ErrorIs(t, joined, errAttachmentTransition, "the synthetic join still carries the sentinel")
250+ verdict, ok := d.routeAttachStopVerdict(attachErr)
251+ require.False(t, ok, "a cleanup-only transition must not decide the verdict")
252+ require.Nil(t, verdict)
253+
254+ verdict, ok = d.routeAttachStopVerdict(errAttachmentTransition)
255+ require.True(t, ok)
256+ var protocolErr *protoErr
257+ require.ErrorAs(t, verdict, &protocolErr)
258+ require.Equal(t, protocol.ErrServerShutdown, protocolErr.code)
259+
260+ d.mu.Lock()
261+ d.closing = false
262+ d.mu.Unlock()
263+ _, ok = d.routeAttachStopVerdict(errAttachmentTransition)
264+ require.False(t, ok, "a transition without an explicit stop is not a shutdown verdict")
265+}
266+
267+// TestFinishRouteAttachSupersessionReportsShutdownVerdict proves the reachable
268+// supersession path: an explicit stop that removes the route's target during the
269+// handshake leaves the attach a leaked transition, and the caller receives the
270+// authoritative shutdown verdict rather than the internal sentinel.
271+func TestFinishRouteAttachSupersessionReportsShutdownVerdict(t *testing.T) {
272+ pty, release := newBlockingPTY(t)
273+ defer release()
274+ d := newTestDaemon(t, newFactory(t, pty), stubClock{})
275+ sess, err := createSessionForTest(d, "work", false, "/tmp/work", defaultSize, terminalEnv{}, d.baseEnv)
276+ require.NoError(t, err)
277+
278+ d.afterAttachmentEffectParticipantsSnapshotted = func(action string, _ []*attachedClient) {
279+ if action != "" {
280+ return
281+ }
282+ d.afterAttachmentEffectParticipantsSnapshotted = nil
283+ // The explicit stop wins the target in the freeze-to-publish window, so
284+ // the attach transition can no longer be published.
285+ d.mu.Lock()
286+ d.closing = true
287+ d.unregisterSessionLocked(sess)
288+ d.mu.Unlock()
289+ }
290+ t.Cleanup(func() { d.afterAttachmentEffectParticipantsSnapshotted = nil })
291+
292+ // The caller must hold d.mu on entry; finishRouteAttach releases it on both
293+ // success and error paths before returning.
294+ d.mu.Lock()
295+ _, err = d.finishRouteAttach(sess, &closeTrackingTransport{}, defaultSize, protocol.Hello{
296+ Version: protocol.Version, Intent: protocol.IntentAttach, Name: "work", Size: defaultSize,
297+ }, true, false)
298+
299+ var protocolErr *protoErr
300+ require.ErrorAs(t, err, &protocolErr)
301+ require.Equal(t, protocol.ErrServerShutdown, protocolErr.code)
302+ d.mu.Lock()
303+ _, retained := d.sessions[sess.id]
304+ d.mu.Unlock()
305+ require.False(t, retained, "the stop owns the target from this point")
306+}
307+
308+// TestFinishRouteAttachKeepsUnrelatedAttachFailure proves the call site keeps an
309+// attach failure that is not a leaked transition, even while the daemon stops and
310+// the rollback cleanup aborts with its own sentinel: only the attach error may
311+// decide the shutdown verdict.
312+func TestFinishRouteAttachKeepsUnrelatedAttachFailure(t *testing.T) {
313+ d, sess, ac, _ := newManualSessionWithPTYs(t, newQuietPTY())
314+
315+ d.afterAttachmentEffectParticipantsSnapshotted = func(action string, _ []*attachedClient) {
316+ if action != "" {
317+ return
318+ }
319+ d.afterAttachmentEffectParticipantsSnapshotted = nil
320+ // The stop begins and a token resume invalidates the rollback's snapshot,
321+ // so the cleanup aborts with a sentinel instead of a clean no-op.
322+ d.mu.Lock()
323+ d.closing = true
324+ d.mu.Unlock()
325+ fresh := &closeTrackingTransport{}
326+ ac.replaceTransport(fresh)
327+ ac.installTestAttachmentCapability(sess.captureAttachmentCapability(ac, fresh))
328+ }
329+ t.Cleanup(func() { d.afterAttachmentEffectParticipantsSnapshotted = nil })
330+
331+ missing := domain.RemoteSessionTarget{LifecycleID: sess.incarnation, SessionName: "work", LiveTabID: "missing-tab"}
332+ d.mu.Lock()
333+ _, err := d.finishRouteAttach(sess, &closeTrackingTransport{}, defaultSize, protocol.Hello{
334+ Version: protocol.Version, Intent: protocol.IntentAttach, Name: "work", Size: defaultSize,
335+ RemoteTarget: &missing, EnvironmentPolicy: protocol.EnvironmentPolicyDaemonOwned,
336+ }, true, false)
337+ var protocolErr *protoErr
338+ require.ErrorAs(t, err, &protocolErr)
339+ require.Equal(t, protocol.ErrNoSuchTarget, protocolErr.code, "an unrelated attach failure must not become a shutdown verdict")
340+ d.mu.Lock()
341+ d.closing = false
342+ d.mu.Unlock()
343+}
M internal/usecase/daemon/render_test.go +1 -1
@@ -886,7 +886,7 @@ func TestResizeOrdersPTYBeforeScreen(t *testing.T) {
886886 require.True(t, gotOutput.Load(), "resize forces a full redraw output")
887887 }
888888
889-// --- reader EOF -> registry-empty shutdown ----------------------------------
889+// --- failed client send teardown --------------------------------------------
890890
891891 func TestSendErrorKeepsEphemeralHeadless(t *testing.T) {
892892 p := portsmocks.NewMockPTY(t)
M internal/usecase/daemon/session.go +159 -27
@@ -470,6 +470,10 @@ func (d *Daemon) createSessionAndSwitch(from *session, ac *attachedClient, name
470470 d.mu.Unlock()
471471 return errors.New("daemon is shutting down")
472472 }
473+ if d.purgeAdmissionClosedLocked() {
474+ d.mu.Unlock()
475+ return errPurgeAdmissionClosed
476+ }
473477 if d.nameLiveOrStoppedLocked(name) {
474478 d.mu.Unlock()
475479 return errSessionNameInUse
@@ -542,6 +546,9 @@ func (d *Daemon) createSessionAndSwitchForAttachment(effect *attachmentEffect, n
542546 if d.closing {
543547 return nil, errors.New("daemon is shutting down")
544548 }
549+ if d.purgeAdmissionClosedLocked() {
550+ return nil, errPurgeAdmissionClosed
551+ }
545552 if d.nameLiveOrStoppedLocked(name) {
546553 return nil, errSessionNameInUse
547554 }
@@ -574,6 +581,10 @@ func (d *Daemon) createEphemeralSessionAndSwitch(from *session, ac *attachedClie
574581 d.mu.Unlock()
575582 return errors.New("daemon is shutting down")
576583 }
584+ if d.purgeAdmissionClosedLocked() {
585+ d.mu.Unlock()
586+ return errPurgeAdmissionClosed
587+ }
577588 from.mu.Lock()
578589 cwd, env := from.cwd, copyEnvironment(from.env)
579590 _, attached := from.attachments[ac]
@@ -620,6 +631,9 @@ func (d *Daemon) createEphemeralSessionAndSwitchForAttachment(effect *attachment
620631 if capability.sess == nil || d.closing {
621632 return nil, errAttachmentTransition
622633 }
634+ if d.purgeAdmissionClosedLocked() {
635+ return nil, errPurgeAdmissionClosed
636+ }
623637 source := capability.sess
624638 source.mu.Lock()
625639 cwd, env := source.cwd, copyEnvironment(source.env)
@@ -1180,8 +1194,7 @@ func (d *Daemon) closeTabLockedWithEffect(sess *session, tb *tab, repaint bool,
11801194 }
11811195 if err != nil {
11821196 d.log.Warn("closing last tab failed", "session", name, "err", err)
1183- d.reportError(sess, domain.UserErr(domain.NoticeSnapshotSaturated,
1184- "couldn't close tab: session state not yet saved; try again", err))
1197+ d.reportError(sess, closeTabFailureNotice(err))
11851198 return err
11861199 }
11871200 d.log.Info("tab closed", "session", name, "last", true)
@@ -1230,6 +1243,19 @@ func (d *Daemon) closeTabLockedWithEffect(sess *session, tb *tab, repaint bool,
12301243 return nil
12311244 }
12321245
1246+// closeTabFailureNotice classifies a failed last-tab close for the attached
1247+// client. A participants-changed abort removed nothing and is safe to retry, so
1248+// it reports that retryable outcome instead of the snapshot-write failure the
1249+// other teardown errors carry.
1250+func closeTabFailureNotice(err error) error {
1251+ if errors.Is(err, errSessionKillParticipantsChanged) {
1252+ return domain.UserErr(domain.NoticeSessionUnavailable,
1253+ "couldn't close tab: session changed during the close; try again", err)
1254+ }
1255+ return domain.UserErr(domain.NoticeSnapshotSaturated,
1256+ "couldn't close tab: session state not yet saved; try again", err)
1257+}
1258+
12331259 // ptyReader drains child output into the VT screen and pokes the dirty channel
12341260 // (non-blocking: a full channel already means a render is pending). On any read
12351261
@@ -1310,6 +1336,15 @@ func (d *Daemon) retryStoppedPurgeContextExact(ctx context.Context, name string,
13101336 return nil
13111337 }
13121338
1339+// killSession removes a session and tears down its resources. It is
1340+// idempotent: only the caller that wins the registry delete acts. Final
1341+// session removal is independent of daemon shutdown: an empty registry leaves
1342+// the daemon serving (closing is untouched) so a later create/restore can
1343+// occupy it again, and only an explicit shutdownAll ends the daemon.
1344+//
1345+// Teardown ordering matters: context cancel and pty.Close run first and
1346+// unconditionally — never gated behind a client send. The Detached notice is
1347+// delivered asynchronously under the drain deadline.
13131348 func (d *Daemon) killSession(sess *session, reason uint8, purge bool) error {
13141349 return d.killSessionWithSnapshotDeadline(sess, reason, purge, nil, nil)
13151350 }
@@ -1408,6 +1443,18 @@ func (d *Daemon) snapshotSessionKillParticipants(target *session, admission *ses
14081443 return snapshot, true
14091444 }
14101445
1446+// sessionKillTargetRemoved reports whether the exact target session has already
1447+// left the live registry, so a post-freeze mismatch is an already-removed no-op
1448+// rather than a participant change that left work undone.
1449+func (d *Daemon) sessionKillTargetRemoved(target *session) bool {
1450+ if target == nil {
1451+ return true
1452+ }
1453+ d.mu.Lock()
1454+ defer d.mu.Unlock()
1455+ return d.sessions[target.id] != target
1456+}
1457+
14111458 // sessionKillParticipantsCurrent repeats lifecycle, source-token, and exact
14121459 // target-membership validation while every discovered attachment gate is frozen.
14131460 func (d *Daemon) sessionKillParticipantsCurrent(snapshot sessionKillParticipants) bool {
@@ -1580,10 +1627,61 @@ func (s *session) finishTeardown() {
15801627 s.teardownMu.Unlock()
15811628 }
15821629
1630+// errSessionKillDeadline reports that a caller-supplied teardown deadline
1631+// expired before killSessionWithSnapshotDeadlineAndCondition froze every
1632+// attachment gate, acquired teardown ownership, or finished joining the
1633+// session's snapshot publication. The session is left live and registered
1634+// exactly as it was: a purge's snapshot quarantine is rolled back, restoring
1635+// eligibility and dirty scheduling for the surviving session, so the abort is
1636+// retry state rather than a lifecycle mutation. It is deliberately distinct
1637+// from a completed teardown: a bounded purge or shutdown must report the skip as
1638+// a typed failure or an incomplete verdict instead of misreporting success.
1639+// Callers that pass no deadline never observe it.
1640+var errSessionKillDeadline = errors.New("session teardown deadline exceeded before ownership")
1641+
1642+// errSessionKillParticipantsChanged reports that a concurrent detach or token
1643+// resume changed the participant set between the pre-freeze snapshot and a
1644+// post-freeze revalidation, so the kill aborted before freezing every gate or
1645+// before publishing terminal ownership. The session was left untouched. It is
1646+// deliberately distinct from a completed teardown so a bounded purge can retry
1647+// against a fresh snapshot instead of counting the abort as a purged unit, and
1648+// so a caller that does not retry still sees that nothing was removed.
1649+var errSessionKillParticipantsChanged = errors.New("session kill participants changed after snapshot")
1650+
1651+// sessionKillRetryMessage is the user-facing, retryable outcome for a teardown
1652+// a concurrent detach or token resume invalidated before it removed anything.
1653+// The sentinel above is internal bookkeeping, so every user-facing path (a
1654+// direct kill or a close-tab command) maps it to this message instead of
1655+// leaking the sentinel text or a permanent-failure verdict.
1656+const sessionKillRetryMessage = "session changed during the operation; try again"
1657+
1658+// teardownDeadlineElapsed reports whether the supplied deadline has already
1659+// expired. A nil deadline never counts as elapsed.
1660+func teardownDeadlineElapsed(deadline *snapshotShutdownDeadline) bool {
1661+ if deadline == nil {
1662+ return false
1663+ }
1664+ select {
1665+ case <-deadline.Done():
1666+ return true
1667+ default:
1668+ return false
1669+ }
1670+}
1671+
15831672 // killSessionWithSnapshotDeadline shares Serve's shutdown budget with its
15841673 // terminal checkpoint and coordinator join. A timed-out repository call keeps
15851674 // only immutable capture state; it never observes closed worker channels or
15861675 // session-owned cache that teardown has released.
1676+//
1677+// A deadline that wins before this call freezes every attachment gate or
1678+// acquires teardown ownership returns errSessionKillDeadline, so a bounded
1679+// caller can distinguish a skip from a completed teardown. An already-removed
1680+// target is a no-op; a target that is still registered while its participant
1681+// set, transport incarnation, or source capability changed after the snapshot
1682+// returns errSessionKillParticipantsChanged, so an aborted teardown is never
1683+// mistaken for a completed one. Ordinary callers that pass nil preserve their
1684+// previous behavior exactly.
15871685 func (d *Daemon) killSessionWithSnapshotDeadline(sess *session, reason uint8, purge bool, deadline *snapshotShutdownDeadline, admission *sessionKillAdmission) error {
15881686 return d.killSessionWithSnapshotDeadlineAndCondition(sess, reason, purge, deadline, admission, false)
15891687 }
@@ -1635,10 +1733,19 @@ func (d *Daemon) killSessionWithSnapshotDeadlineAndCondition(sess *session, reas
16351733 // The acquisition helper has already rolled back exactly the gates it
16361734 // owned. Do not enter architecture locks or publish terminal ownership
16371735 // changes unless the complete canonical participant set is frozen.
1736+ if teardownDeadlineElapsed(deadline) {
1737+ return errSessionKillDeadline
1738+ }
16381739 return nil
16391740 }
16401741 if !d.sessionKillParticipantsCurrent(participants) {
1641- return nil
1742+ if d.sessionKillTargetRemoved(sess) {
1743+ // The exact target already left the registry, so there is nothing left
1744+ // to tear down: an already-removed session is a no-op, not a
1745+ // participant change.
1746+ return nil
1747+ }
1748+ return errSessionKillParticipantsChanged
16421749 }
16431750
16441751 // Pane EOF, explicit close, and daemon shutdown can all converge on the same
@@ -1649,9 +1756,28 @@ func (d *Daemon) killSessionWithSnapshotDeadlineAndCondition(sess *session, reas
16491756 var acquired bool
16501757 reason, purge, acquired = sess.beginTeardownRequest(deadline, reason, purge)
16511758 if !acquired {
1759+ // beginTeardownRequest only fails to acquire when the deadline expired
1760+ // (without one it waits until ownership is available), so the session is
1761+ // still untouched and the skip must stay observable.
1762+ if deadline != nil {
1763+ return errSessionKillDeadline
1764+ }
16521765 return nil
16531766 }
16541767 defer sess.finishTeardown()
1768+ // Capture eligibility before any coordinator quarantine flips it. Every
1769+ // post-quarantine abort that leaves this session registered restores exactly
1770+ // this value, so a surviving session resumes checkpoint scheduling instead of
1771+ // staying permanently frozen. The rollback is gated on the same teardown
1772+ // ownership as finishTeardown and on the exact registry entry, so a completed
1773+ // removal never resurrects scheduling.
1774+ wasEligible := sess.snapEligible.Load()
1775+ quarantined := false
1776+ defer func() {
1777+ if quarantined && d.sessionRegistered(sess) {
1778+ rollbackSnapshotQuarantine(sess, wasEligible)
1779+ }
1780+ }()
16551781
16561782 d.mu.Lock()
16571783 current := d.sessions[sess.id]
@@ -1673,13 +1799,26 @@ func (d *Daemon) killSessionWithSnapshotDeadlineAndCondition(sess *session, reas
16731799 incarnation := sess.incarnation
16741800 sess.mu.Unlock()
16751801 // Join snapshot publication before changing live identity or deleting the
1676- // incarnation directory, so an older publication cannot recreate it.
1802+ // incarnation directory, so an older publication cannot recreate it. A shared
1803+ // caller deadline bounds the join: a repository call that ignores cancellation
1804+ // must never hold purge admission open, so an expired budget leaves the unit
1805+ // live and reports the typed skip instead.
16771806 if purge && !isEphemeral {
16781807 if err := d.beginSnapshotPurge(name, incarnation); err != nil {
16791808 return err
16801809 }
16811810 if d.snapshotRepository != nil {
1682- <-quarantineSnapshotCoordinator(sess)
1811+ quarantineDone := quarantineSnapshotCoordinator(sess)
1812+ quarantined = true
1813+ if deadline != nil {
1814+ select {
1815+ case <-quarantineDone:
1816+ case <-deadline.Done():
1817+ return errSessionKillDeadline
1818+ }
1819+ } else {
1820+ <-quarantineDone
1821+ }
16831822 }
16841823 }
16851824 if !purge && !isEphemeral && d.persistEnabled {
@@ -1719,6 +1858,7 @@ func (d *Daemon) killSessionWithSnapshotDeadlineAndCondition(sess *session, reas
17191858 } else {
17201859 quarantineDone = quarantineSnapshotCoordinator(sess)
17211860 }
1861+ quarantined = true
17221862 joined := true
17231863 if deadline != nil {
17241864 select {
@@ -1734,6 +1874,9 @@ func (d *Daemon) killSessionWithSnapshotDeadlineAndCondition(sess *session, reas
17341874 sess.snapshotChunkCache = nil
17351875 sess.snapshotMu.Unlock()
17361876 }
1877+ if d.afterSessionKillQuarantine != nil {
1878+ d.afterSessionKillQuarantine(sess)
1879+ }
17371880 }
17381881 // Revalidate the exact source capability, target lifecycle, and attachment
17391882 // memberships under canonical architecture locks immediately before
@@ -1757,7 +1900,7 @@ func (d *Daemon) killSessionWithSnapshotDeadlineAndCondition(sess *session, reas
17571900 unlockSessions()
17581901 d.notices.routingMu.Unlock()
17591902 d.mu.Unlock()
1760- return nil
1903+ return errSessionKillParticipantsChanged
17611904 }
17621905
17631906 // d.mu -> routingMu -> ordered sessions -> ordered coordinators are held
@@ -1798,13 +1941,6 @@ func (d *Daemon) killSessionWithSnapshotDeadlineAndCondition(sess *session, reas
17981941 stopped.purging = purge
17991942 d.inactive[stoppedName] = stopped
18001943 }
1801- empty := len(d.sessions) == 0
1802- if empty {
1803- // Shutdown is now inevitable and irreversible (doneOnce below): stop
1804- // route from inserting new sessions from this instant, while we still
1805- // hold the same lock route checks under.
1806- d.closing = true
1807- }
18081944 d.mu.Unlock()
18091945 d.finishParkedAttachmentRetirements(parkedRetirements)
18101946 frozen.unfreeze()
@@ -1844,27 +1980,23 @@ func (d *Daemon) killSessionWithSnapshotDeadlineAndCondition(sess *session, reas
18441980 }
18451981 // The coordinator and all panes are now stopped, so no producer can publish
18461982 // another generation after this destructive source deletion. Keep the hidden
1847- // stopped record when deletion fails so a repeated live kill can retry.
1983+ // stopped record when deletion fails or is still pending so a repeated live
1984+ // kill can retry. The purge-owned exact delete is bounded: a repository
1985+ // delete that ignores cancellation is detached behind a name/incarnation/
1986+ // createdAt fence rather than holding purgeAllMu, admission, or control
1987+ // responses.
18481988 if !ephemeral && purge {
1849- err := d.finishSnapshotPurge(d.serveCtx, stoppedName, incarnation, stoppedRecord.CreatedAt)
1850- if err != nil {
1989+ if err := d.finishLiveDurablePurge(deadline, stoppedRecord); err != nil {
18511990 purgeErr = errors.Join(purgeErr, err)
18521991 d.log.Warn("finishing session snapshot purge failed", "err", err, "session", stoppedName)
1853- } else {
1854- expected := inactiveSessionFromRecord(stoppedRecord, protocol.SessionDown, nil)
1855- d.mu.Lock()
1856- if stopped, ok := d.inactive[stoppedName]; ok && stopped.purging && stopped.sameLifecycle(expected) {
1857- delete(d.inactive, stoppedName)
1858- }
1859- d.mu.Unlock()
18601992 }
18611993 }
1862- if purge && purgeErr == nil {
1994+ if purge && purgeErr == nil && isEphemeral {
1995+ // An ephemeral purge owns no durable record, but the remaining sessions'
1996+ // route histories may still reference it.
18631997 d.reconcileAllRouteHistories()
18641998 }
1865- if empty {
1866- d.doneOnce.Do(func() { close(d.done) })
1867- } else if ringing {
1999+ if ringing {
18682000 d.repaintAllAttachedClients()
18692001 }
18702002
M internal/usecase/daemon/session_test.go +208 -56
@@ -136,16 +136,51 @@ func TestHandshakeNoSuchSession(t *testing.T) {
136136 require.Equal(t, protocol.ErrNoSuchSession, em.Code)
137137 }
138138
139-func TestKillAllEmptyDaemonSignalsShutdown(t *testing.T) {
140- d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), stubClock{})
139+// TestKillAllEmptyDaemonKeepsServing proves KillAll no longer ends the daemon:
140+// purging an empty registry leaves closing, move admission, the daemon-wide
141+// pane process context, and d.done untouched, and the same daemon then admits a
142+// fresh session.
143+func TestKillAllEmptyDaemonKeepsServing(t *testing.T) {
144+ pty, release := newBlockingPTY(t)
145+ defer release()
146+ d := newTestDaemon(t, newFactory(t, pty), stubClock{})
141147 tr, _, _ := newConn(t, mustClientEnvelope(protocol.Kill{Scope: protocol.KillAll}))
142148 d.handleConn(tr)
143149
144150 select {
145151 case <-d.done:
146- case <-time.After(time.Second):
147- t.Fatal("kill all on empty daemon did not signal shutdown")
152+ t.Fatal("kill all on empty daemon must not signal shutdown")
153+ default:
148154 }
155+ d.mu.Lock()
156+ closing := d.closing
157+ d.mu.Unlock()
158+ require.False(t, closing, "kill all must not mark the daemon closing")
159+ d.moveLifecycleMu.Lock()
160+ moveClosing := d.moveLifecycleClosing
161+ d.moveLifecycleMu.Unlock()
162+ require.False(t, moveClosing, "kill all must not close move admission")
163+ select {
164+ case <-d.paneProcessCtx.Done():
165+ t.Fatal("kill all must not cancel the daemon-wide pane process context")
166+ default:
167+ }
168+
169+ // The purge reopened admission: the same daemon creates a fresh session.
170+ tr2, sends2, releaseConn2 := newConn(t, mustHello(protocol.IntentEphemeral, "", domain.Size{Cols: 80, Rows: 24}))
171+ var hg sync.WaitGroup
172+ hg.Go(func() { d.handleConn(tr2) })
173+ welcome := decodeServerMessage(t, awaitFrame(t, sends2, "Welcome")).(protocol.Welcome)
174+ require.Equal(t, "0", welcome.SessionName)
175+ awaitFrame(t, sends2, "Output")
176+ require.Equal(t, 1, sessionCount(d))
177+
178+ releaseConn2()
179+ _ = d.killSession(firstSession(d), protocol.ReasonSessionKilled, true)
180+ release()
181+ hg.Wait()
182+ d.sessWg.Wait()
183+ d.waitNotifies()
149184 }
150185
151186 func TestBeginTeardownRequestResolvesShutdownPreservation(t *testing.T) {
@@ -382,14 +417,21 @@ func TestDetachKeepsNamed(t *testing.T) {
382417
383418 // --- render scheduler debounce ----------------------------------------------
384419
385-func TestReaderEOFRemovesSessionAndSignalsShutdown(t *testing.T) {
386- p := portsmocks.NewMockPTY(t)
420+// TestReaderEOFRemovesSessionAndDaemonSurvives proves final session removal is
421+// independent of daemon shutdown: the last session's EOF empties the registry
422+// without signalling shutdown or marking the daemon closing, and the same
423+// daemon can be occupied again by a fresh session.
424+func TestReaderEOFRemovesSessionAndDaemonSurvives(t *testing.T) {
425+ eofPTY := portsmocks.NewMockPTY(t)
387426 // Read returns EOF immediately (child already gone).
388- p.EXPECT().Read(mock.Anything).Return(0, io.EOF).Maybe()
389- p.EXPECT().Close().Return(nil).Maybe()
390- p.EXPECT().Pid().Return(1).Maybe()
427+ eofPTY.EXPECT().Read(mock.Anything).Return(0, io.EOF).Maybe()
428+ eofPTY.EXPECT().Close().Return(nil).Maybe()
429+ eofPTY.EXPECT().Pid().Return(1).Maybe()
430+
431+ secondPTY, releaseSecond := newBlockingPTY(t)
432+ defer releaseSecond()
433+ d := newTestDaemon(t, newFactorySeq(t, eofPTY, secondPTY), stubClock{})
391434
392- d := newTestDaemon(t, newFactory(t, p), stubClock{})
393435 tr, sends, _ := newConn(t, mustHello(protocol.IntentEphemeral, "", domain.Size{Cols: 80, Rows: 24}))
394436
395437 var hg sync.WaitGroup
@@ -400,27 +442,140 @@ func TestReaderEOFRemovesSessionAndSignalsShutdown(t *testing.T) {
400442 dm := decodeServerMessage(t, det).(protocol.Detached)
401443 require.Equal(t, protocol.ReasonSessionKilled, dm.Reason)
402444
445+ // The registry is empty, yet the daemon has not begun shutdown and has not
446+ // signalled Serve. The Detached notice is scheduled only after registry
447+ // removal has published, so these assertions are deterministic.
403448 select {
404449 case <-d.done:
405- case <-time.After(2 * time.Second):
406- t.Fatal("registry-empty shutdown was not signalled")
450+ t.Fatal("final session removal must not signal daemon shutdown")
451+ default:
407452 }
453+ d.mu.Lock()
454+ closing := d.closing
455+ d.mu.Unlock()
456+ require.False(t, closing, "empty registry must not mark the daemon closing")
408457 require.Equal(t, 0, sessionCount(d))
409458
459+ // The empty registry can be occupied again by a brand-new session.
460+ tr2, sends2, releaseConn2 := newConn(t, mustHello(protocol.IntentEphemeral, "", domain.Size{Cols: 80, Rows: 24}))
461+ var hg2 sync.WaitGroup
462+ hg2.Go(func() { d.handleConn(tr2) })
463+ welcome := decodeServerMessage(t, awaitFrame(t, sends2, "Welcome")).(protocol.Welcome)
464+ require.Equal(t, "0", welcome.SessionName)
465+ awaitFrame(t, sends2, "Output")
466+ require.Equal(t, 1, sessionCount(d))
467+
468+ _ = d.killSession(firstSession(d), protocol.ReasonSessionKilled, true)
469+ releaseConn2()
410470 hg.Wait()
471+ hg2.Wait()
472+ d.sessWg.Wait()
473+ d.waitNotifies()
474+}
475+
476+// TestConcurrentFinalRemovalAndCreationLinearize proves final-session removal
477+// no longer opens a shutdown window: a create racing the removal is always
478+// admitted (never rejected with ErrServerShutdown) and the two registry
479+// transitions linearize under d.mu, leaving exactly the newly created session.
480+func TestConcurrentFinalRemovalAndCreationLinearize(t *testing.T) {
481+ for attempt := 0; attempt < 32; attempt++ {
482+ firstPTY, releaseFirst := newBlockingPTY(t)
483+ secondPTY, releaseSecond := newBlockingPTY(t)
484+ d := newTestDaemon(t, newFactorySeq(t, firstPTY, secondPTY), stubClock{})
485+
486+ sess, err := createSessionForTest(d, "0", true, "/tmp", domain.Size{Cols: 80, Rows: 24}, terminalEnv{}, nil)
487+ require.NoError(t, err)
488+
489+ start := make(chan struct{})
490+ var wg sync.WaitGroup
491+ wg.Add(2)
492+ var createErr error
493+ go func() {
494+ defer wg.Done()
495+ <-start
496+ _, _, createErr = d.route(helloResumeCapable(protocol.IntentEphemeral, "", 0), &closeTrackingTransport{})
497+ }()
498+ go func() {
499+ defer wg.Done()
500+ <-start
501+ _ = d.killSession(sess, protocol.ReasonSessionKilled, true)
502+ }()
503+ close(start)
504+ wg.Wait()
505+
506+ require.NoError(t, createErr, "attempt %d: creation racing final removal must be admitted", attempt)
507+ require.Equal(t, 1, sessionCount(d), "attempt %d: exactly the new session must remain", attempt)
508+ d.mu.Lock()
509+ closing := d.closing
510+ d.mu.Unlock()
511+ require.False(t, closing, "attempt %d: empty→occupied must not mark the daemon closing", attempt)
512+
513+ _ = d.killSession(firstSession(d), protocol.ReasonSessionKilled, true)
514+ releaseFirst()
515+ releaseSecond()
516+ d.sessWg.Wait()
517+ d.waitNotifies()
518+ }
519+}
520+
521+// TestRestoreAfterEmptyRegistryWorks proves a stopped named session can be
522+// restored by a fresh attach once the registry has returned to empty: the
523+// daemon kept serving and the inactive authority survived the final removal.
524+func TestRestoreAfterEmptyRegistryWorks(t *testing.T) {
525+ firstPTY, releaseFirst := newBlockingPTY(t)
526+ defer releaseFirst()
527+ secondPTY, releaseSecond := newBlockingPTY(t)
528+ defer releaseSecond()
529+ d := newTestDaemon(t, newFactorySeq(t, firstPTY, secondPTY), stubClock{})
530+
531+ sess, err := createSessionForTest(d, "work", false, t.TempDir(), domain.Size{Cols: 80, Rows: 24}, terminalEnv{}, d.baseEnv)
532+ require.NoError(t, err)
533+
534+ // Stop the only session; the registry returns to empty without shutdown.
535+ require.NoError(t, d.killSession(sess, protocol.ReasonServerShutdown, false))
536+ require.Equal(t, 0, sessionCount(d))
537+ select {
538+ case <-d.done:
539+ t.Fatal("stopped final session must not signal daemon shutdown")
540+ default:
541+ }
542+
543+ // Restore it inside the same daemon.
544+ tr := newMockServerConnection(t)
545+ tr.EXPECT().Send(mock.Anything).Return(nil).Maybe()
546+ tr.EXPECT().Close().Return(nil).Maybe()
547+ restored, ac, err := d.route(protocol.Hello{Version: protocol.Version, Intent: protocol.IntentAttach, Name: "work", Size: domain.Size{Cols: 80, Rows: 24}}, tr)
548+ require.NoError(t, err)
549+ require.NotNil(t, ac)
550+ require.Equal(t, "work", restored.name)
551+ require.Equal(t, 1, sessionCount(d))
552+
553+ _ = d.killSession(restored, protocol.ReasonServerShutdown, false)
411554 d.sessWg.Wait()
412555 d.waitNotifies()
413556 }
414557
415558 // --- full Serve lifecycle ---------------------------------------------------
416559
417-func TestServeReturnsWhenLastSessionExits(t *testing.T) {
418- p, releasePTY := newBlockingPTY(t)
419- tr, sends, _ := newConn(t, mustHello(protocol.IntentEphemeral, "", domain.Size{Cols: 80, Rows: 24}))
560+// TestServeSurvivesEmptyRegistryAndAcceptsNewSession proves the accept loop and
561+// listener are not torn down by an empty registry: after the last session
562+// exits, the same Serve loop still admits a brand-new session, and only ctx
563+// cancellation ends it.
564+func TestServeSurvivesEmptyRegistryAndAcceptsNewSession(t *testing.T) {
565+ firstPTY, releaseFirst := newBlockingPTY(t)
566+ defer releaseFirst()
567+ secondPTY, releaseSecond := newBlockingPTY(t)
568+ defer releaseSecond()
569+
570+ firstTr, firstSends, _ := newConn(t, mustHello(protocol.IntentEphemeral, "", domain.Size{Cols: 80, Rows: 24}))
571+ secondTr, secondSends, _ := newConn(t, mustHello(protocol.IntentEphemeral, "", domain.Size{Cols: 80, Rows: 24}))
420572
421573 l := newMockServerListener(t)
574+ // Feed connections one at a time: the second is offered only after the
575+ // first session has exited, so the mock factory's PTY order is deterministic
576+ // and the reuse handshake proves the accept loop survived.
422577 connCh := make(chan wire.Transport, 1)
423- connCh <- tr
578+ connCh <- firstTr
424579 closed := make(chan struct{})
425580 var once sync.Once
426581 l.EXPECT().Accept().RunAndReturn(func() (wire.Transport, error) {
@@ -434,22 +589,35 @@ func TestServeReturnsWhenLastSessionExits(t *testing.T) {
434589 l.EXPECT().Close().RunAndReturn(func() error { once.Do(func() { close(closed) }); return nil }).Maybe()
435590 l.EXPECT().Addr().Return("mock").Maybe()
436591
437- d := New(newFactory(t, p), stubClock{}, slog.New(slog.NewTextHandler(io.Discard, nil)))
592+ d := New(newFactorySeq(t, firstPTY, secondPTY), stubClock{}, slog.New(slog.NewTextHandler(io.Discard, nil)))
593+ ctx, cancel := context.WithCancel(context.Background())
594+ defer cancel()
438595
439596 served := make(chan error, 1)
440- go func() { served <- d.Serve(context.Background(), l) }()
597+ go func() { served <- d.Serve(ctx, l) }()
441598
442- awaitFrame(t, sends, "Welcome")
443- awaitFrame(t, sends, "Output")
599+ awaitFrame(t, firstSends, "Welcome")
600+ awaitFrame(t, firstSends, "Output")
444601
445- // Child exits -> session removed -> registry empties -> Serve returns.
446- releasePTY()
602+ // Child exits -> session removed -> registry empties -> Serve must survive.
603+ releaseFirst()
604+ det := awaitFrame(t, firstSends, "Detached")
605+ require.Equal(t, protocol.ReasonSessionKilled, decodeServerMessage(t, det).(protocol.Detached).Reason)
606+ require.Equal(t, 0, sessionCount(d))
607+
608+ // The same accept loop admits a fresh connection into the empty registry.
609+ connCh <- secondTr
610+ awaitFrame(t, secondSends, "Welcome")
611+ awaitFrame(t, secondSends, "Output")
612+ require.Equal(t, 1, sessionCount(d))
447613
614+ // Only explicit cancellation stops Serve.
615+ cancel()
448616 select {
449617 case err := <-served:
450618 require.NoError(t, err)
451619 case <-time.After(3 * time.Second):
452- t.Fatal("Serve did not return after last session exited")
620+ t.Fatal("Serve did not return after context cancel")
453621 }
454622 }
455623
@@ -499,22 +667,18 @@ func TestServeGracefulShutdownOnContextCancel(t *testing.T) {
499667
500668 // --- shutdown/create interlock ------------------------------------------------
501669
502-// TestHelloRacingShutdownIsRejected covers the interlock between the
503-// registry-empty shutdown decision and session creation. In particular, the
504-// old `.Once()` factory expectation was wrong: firstPaint legitimately queues
505-// a floating prewarm, so an Open that starts before killSession is valid work,
506-// not a leaked post-shutdown child. The factory markers classify timing at the
507-// Open boundary: this test first observes that valid prewarm, then verifies
508-// that d.done (closed synchronously by killSession) is never followed by an
509-// Open start.
510-func TestHelloRacingShutdownIsRejected(t *testing.T) {
670+// TestHelloAfterExplicitShutdownIsRejected covers the shutdown/create
671+// interlock that survives P4.1: an explicit shutdownAll still marks the daemon
672+// closing synchronously and signals Serve, so Hellos arriving afterward are
673+// rejected with a clean typed error instead of creating a session the teardown
674+// snapshot missed. Final session removal alone no longer triggers this.
675+func TestHelloAfterExplicitShutdownIsRejected(t *testing.T) {
511676 p, releasePTY := newBlockingPTY(t)
512677 floating := newQuietPTY()
513678 f := portsmocks.NewMockPTYFactory(t)
514679 normalSize := domain.Size{Cols: 80, Rows: 22}
515- preShutdownFloatingOpen := make(chan struct{}, 1)
680+ var d *Daemon
516681 var opensAfterShutdown atomic.Int32
517- d := newTestDaemon(t, f, stubClock{})
518682 f.EXPECT().Open(mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).RunAndReturn(
519683 func(_ context.Context, _ string, _ []string, _ []string, _ string, geometry domain.Geometry) (ports.PTY, error) {
520684 select {
@@ -525,40 +689,29 @@ func TestHelloRacingShutdownIsRejected(t *testing.T) {
525689 if geometry.Size == normalSize {
526690 return p, nil
527691 }
528- select {
529- case preShutdownFloatingOpen <- struct{}{}:
530- default:
531- }
532692 return floating, nil
533693 },
534694 ).Maybe()
695+ d = newTestDaemon(t, f, stubClock{})
535696
536697 tr1, sends1, release1 := newConn(t, mustHello(protocol.IntentEphemeral, "", domain.Size{Cols: 80, Rows: 24}))
537698 var hg sync.WaitGroup
538699 hg.Go(func() { d.handleConn(tr1) })
539700 awaitFrame(t, sends1, "Welcome")
540- // firstPaint starts the background prewarm asynchronously. It is valid for
541- // this Open to occur before shutdown, so do not mistake it for the racing
542- // Hello's forbidden launch.
543701 awaitFrame(t, sends1, "Output")
544- select {
545- case <-preShutdownFloatingOpen:
546- case <-time.After(time.Second):
547- t.Fatal("legitimate pre-shutdown floating prewarm did not start")
548- }
549- sess := firstSession(d)
550- require.NotNil(t, sess)
702+ require.Equal(t, 1, sessionCount(d))
551703
552- // The last session dies: shutdown begins irreversibly.
553- _ = d.killSession(sess, protocol.ReasonSessionKilled, false)
704+ // Explicit shutdown begins irreversibly and signals Serve synchronously.
705+ require.False(t, d.shutdownAll(protocol.ReasonServerShutdown))
554706 select {
555707 case <-d.done:
556708 default:
557- t.Fatal("registry-empty shutdown must be signalled synchronously by killSession")
709+ t.Fatal("explicit shutdown must signal Serve synchronously")
558710 }
711+ require.Equal(t, 0, sessionCount(d))
559712
560- // Racing Hellos — both creation intents must be rejected with a clean
561- // typed error, and nothing may be inserted into the registry.
713+ // Hellos after shutdown — both creation intents — must be rejected with a
714+ // clean typed error, and nothing may be inserted into the registry.
562715 for _, intent := range []struct {
563716 name string
564717 intent uint8
@@ -572,17 +725,16 @@ func TestHelloRacingShutdownIsRejected(t *testing.T) {
572725 e := awaitFrame(t, sends2, "Error")
573726 em, ok := decodeServerMessage(t, e).(protocol.ErrorMsg)
574727 require.True(t, ok)
575- require.Equal(t, protocol.ErrServerShutdown, em.Code, "%s hello racing shutdown must be rejected", intent.name)
728+ require.Equal(t, protocol.ErrServerShutdown, em.Code, "%s hello after shutdown must be rejected", intent.name)
576729 }
577730 require.Equal(t, 0, sessionCount(d), "no session may be inserted after shutdown began")
578- require.Zero(t, opensAfterShutdown.Load(), "no PTYFactory.Open may start after d.done/shutdown completion")
579731
580732 release1()
581733 releasePTY()
582734 hg.Wait()
583735 d.sessWg.Wait()
584736 d.waitNotifies()
585- require.Zero(t, opensAfterShutdown.Load(), "racing Hellos must not start a PTY after shutdown")
737+ require.Zero(t, opensAfterShutdown.Load(), "shutdown must not be followed by a PTY launch")
586738 }
587739
588740 // --- wedged-client teardown ---------------------------------------------------
M internal/usecase/daemon/snapshot_coordinator.go +32 -1
@@ -50,6 +50,32 @@ func quarantineSnapshotCoordinatorWithOptions(sess *session, retainQueuedCapture
5050 return done
5151 }
5252
53+// rollbackSnapshotQuarantine reverses quarantineSnapshotCoordinator when a
54+// teardown aborts before it owns any destructive repository operation and the
55+// session therefore stays live and registered. The quarantine cancelled the
56+// publication context and flipped eligibility, so a surviving session would
57+// otherwise never checkpoint again: this discards the cancelled context so the
58+// next schedule allocates a fresh one, restores the captured pre-quarantine
59+// eligibility, and marks the session dirty so the mutations its ineligible fast
60+// path suppressed during the quarantine window are captured by a later
61+// publication. A queued capture the quarantine discarded stays discarded:
62+// startSnapshotPublication rejects it by identity and the forced dirty
63+// generation guarantees a replacement capture.
64+func rollbackSnapshotQuarantine(sess *session, wasEligible bool) {
65+ if sess == nil {
66+ return
67+ }
68+ sess.snapshotMu.Lock()
69+ sess.snapshotQuarantined = false
70+ sess.snapshotPublicationContext = nil
71+ sess.snapshotPublicationCancel = nil
72+ sess.snapshotMu.Unlock()
73+ sess.snapEligible.Store(wasEligible)
74+ if wasEligible {
75+ markSnapshotDirty(sess)
76+ }
77+}
78+
5379 func markSnapshotDirty(sess *session) {
5480 if sess == nil || !sess.snapEligible.Load() {
5581 return
@@ -160,7 +186,12 @@ func (d *Daemon) finishSnapshotCapture(capture *snapshotCapture, succeeded bool)
160186 capture.finishOnce.Do(func() {
161187 var shouldScheduleForcedSuccessor bool
162188 capture.session.snapshotMu.Lock()
163- if capture.session.snapshotPendingCaptures > 0 {
189+ // A capture the quarantine discarded already dropped its pending count when
190+ // it was detached from the producer queue; the worker still calls finish on
191+ // it after rejecting the stale identity. Decrementing again here would
192+ // under-count a replacement capture scheduled after a rollback, so a
193+ // discarded capture must not decrement twice.
194+ if !capture.coordinatorDiscarded && capture.session.snapshotPendingCaptures > 0 {
164195 capture.session.snapshotPendingCaptures--
165196 }
166197 if capture.session.snapshotQueuedCapture == capture {
M internal/usecase/daemon/snapshot_restore.go +7 -4
@@ -109,7 +109,7 @@ func validateRestoreSessionSnapshot(snap snapcodec.Session) error {
109109
110110 // restoreSession owns the restored tabs until registration succeeds. Every
111111 // unsuccessful path closes their PTYs and cancels their shared session context.
112-func (d *Daemon) restoreSession(ctx context.Context, snap snapcodec.Session, repositoryGeneration uint64, checkpoint domain.CheckpointRef) error {
112+func (d *Daemon) restoreSession(ctx context.Context, snap snapcodec.Session, repositoryGeneration uint64, checkpoint domain.CheckpointRef, epoch uint64) error {
113113 if err := ctx.Err(); err != nil {
114114 return err
115115 }
@@ -147,7 +147,7 @@ func (d *Daemon) restoreSession(ctx context.Context, snap snapcodec.Session, rep
147147 // checkpoints must continue from it rather than reuse generation one.
148148 sess.snapshotPublishedGeneration = repositoryGeneration
149149 sess.snapshotPublishedCheckpoint = &checkpoint
150- registered, err := d.persistAndRegisterRestoredSession(ctx, sess)
150+ registered, err := d.persistAndRegisterRestoredSession(ctx, sess, epoch)
151151 if err != nil || !registered {
152152 return err
153153 }
@@ -313,7 +313,10 @@ func (d *Daemon) newRestoredSession(snap snapcodec.Session, sctx context.Context
313313
314314 // persistAndRegisterRestoredSession atomically transfers a successfully
315315 // persisted session to the daemon. The caller retains cleanup on false/error.
316-func (d *Daemon) persistAndRegisterRestoredSession(ctx context.Context, sess *session) (bool, error) {
316+// The epoch revalidation makes publication a true admitted transition: if a
317+// KillAll purge began and is waiting on this admitted restore, the session is
318+// not published, so it cannot resurrect a record the purge is about to remove.
319+func (d *Daemon) persistAndRegisterRestoredSession(ctx context.Context, sess *session, epoch uint64) (bool, error) {
317320 if sess.incarnation == (domain.IncarnationID{}) {
318321 d.mu.Lock()
319322 stopped := d.inactive[sess.name]
@@ -343,7 +346,7 @@ func (d *Daemon) persistAndRegisterRestoredSession(ctx context.Context, sess *se
343346 }
344347 d.mu.Lock()
345348 defer d.mu.Unlock()
346- if d.closing || d.findByNameLocked(sess.name) != nil || ctx.Err() != nil {
349+ if d.purgeSupersededLocked(epoch) || d.findByNameLocked(sess.name) != nil || ctx.Err() != nil {
347350 return false, nil
348351 }
349352 sess.id = domain.SessionID(fmt.Sprintf("sess-%d", d.nextID))
M internal/usecase/daemon/snapshot_restore_incremental.go +62 -12
@@ -39,26 +39,64 @@ func (d *Daemon) restoreIncrementalSnapshots(ctx context.Context) {
3939 if d.catalogue == nil {
4040 return
4141 }
42+ records, epoch, ok := d.captureCatalogueSnapshot(ctx)
43+ if !ok {
44+ return
45+ }
46+ d.restoreCatalogueEpoch(ctx, records, epoch)
47+}
48+
49+// captureCatalogueSnapshot reads the purge boundary and the authoritative
50+// catalogue records together under purge admission. Holding the gate for both
51+// reads keeps them atomic with any KillAll: a purge can never increment the
52+// epoch between the two reads and pair a record set it already removed with a
53+// boundary that looks current. The caller's context and the move/purge gates
54+// bound the wait.
55+func (d *Daemon) captureCatalogueSnapshot(ctx context.Context) ([]domain.CatalogueRecord, uint64, bool) {
56+ if err := d.acquirePurgeAdmission(ctx); err != nil {
57+ return nil, 0, false
58+ }
59+ defer d.releasePurgeAdmission()
60+ epoch := d.purgeEpochSnapshot()
4261 records, err := d.catalogue.Records()
4362 if err != nil {
4463 d.log.Error("loading catalogue for snapshot restoration failed", "err", err)
45- return
64+ return nil, 0, false
4665 }
47- d.restoreCatalogue(ctx, records)
66+ return records, epoch, true
4867 }
4968
50-// restoreCatalogue gives every expected record an independent bounded job.
51-// A failed record is degraded without cancelling any sibling restoration.
52-func (d *Daemon) restoreCatalogue(ctx context.Context, records []domain.CatalogueRecord) {
69+// restoreCatalogueEpoch restores records captured at the given purge boundary.
70+// Each worker holds purge admission for exactly one record's transition — its
71+// registry publication, checkpoint load, and publication or demotion — so a
72+// KillAll drains the in-flight record before it snapshots, and a worker that
73+// waited on admission abandons records superseded by a completed purge. Holding
74+// admission per record rather than per worker keeps the purge's drain latency
75+// proportional to the slowest single record, not the whole catalogue.
76+func (d *Daemon) restoreCatalogueEpoch(ctx context.Context, records []domain.CatalogueRecord, epoch uint64) {
5377 jobs := make(chan domain.CatalogueRecord)
5478 workers := min(catalogueRestoreConcurrency, len(records))
5579 var wg sync.WaitGroup
5680 for range workers {
5781 wg.Go(func() {
5882 for record := range jobs {
59- done := d.ensureCatalogueRegistryEntry(record)
60- err := d.restoreRecord(ctx, record)
83+ if err := d.acquirePurgeAdmission(ctx); err != nil {
84+ // Admission is closed for good (shutdown or cancellation): drain
85+ // the queue so the producer can finish, and touch no registry state.
86+ continue
87+ }
88+ if ctx.Err() != nil {
89+ d.releasePurgeAdmission()
90+ continue
91+ }
92+ done, ok := d.ensureCatalogueRegistryEntry(record, epoch)
93+ if !ok {
94+ d.releasePurgeAdmission()
95+ continue
96+ }
97+ err := d.restoreRecord(ctx, record, epoch)
6198 d.finishRecordRestore(record, err, done)
99+ d.releasePurgeAdmission()
62100 }
63101 })
64102 }
@@ -100,15 +138,22 @@ func inactiveSessionFromRecord(record domain.CatalogueRecord, state protocol.Ses
100138 }
101139 }
102140
103-func (d *Daemon) ensureCatalogueRegistryEntry(record domain.CatalogueRecord) chan struct{} {
141+// ensureCatalogueRegistryEntry publishes a stopped registry entry for a record
142+// captured at epoch, unless a purge or explicit shutdown has superseded it. The
143+// false result means the record must be skipped: it no longer belongs to the
144+// authoritative set, so creating an entry here would resurrect it.
145+func (d *Daemon) ensureCatalogueRegistryEntry(record domain.CatalogueRecord, epoch uint64) (chan struct{}, bool) {
104146 d.mu.Lock()
105147 defer d.mu.Unlock()
148+ if d.purgeSupersededLocked(epoch) {
149+ return nil, false
150+ }
106151 if entry, ok := d.inactive[record.Name]; ok {
107- return entry.restoreDone
152+ return entry.restoreDone, true
108153 }
109154 state, done := initialSessionState(record)
110155 d.inactive[record.Name] = inactiveSessionFromRecord(record, state, done)
111- return done
156+ return done, true
112157 }
113158
114159 // closeRuntimeRestoreDoneLocked closes a per-record restoration barrier.
@@ -173,7 +218,12 @@ func (d *Daemon) finishRecordRestore(record domain.CatalogueRecord, restoreErr e
173218 }
174219 }
175220
176-func (d *Daemon) restoreRecord(ctx context.Context, record domain.CatalogueRecord) error {
221+func (d *Daemon) restoreRecord(ctx context.Context, record domain.CatalogueRecord, epoch uint64) error {
222+ if d.purgeSuperseded(epoch) {
223+ // A purge or shutdown owns this record now; skip the load entirely so no
224+ // PTY is opened for a lifecycle that will not survive.
225+ return nil
226+ }
177227 if record.DegradedReason != "" {
178228 d.setStoppedRecovery(record, protocol.SessionBroken)
179229 d.logSessionDegraded(record, "persisted-broken")
@@ -229,7 +279,7 @@ func (d *Daemon) restoreRecord(ctx context.Context, record domain.CatalogueRecor
229279 d.log.Info("snapshot_checkpoint_reconciliation_complete", "session", record.Name, "incarnation", record.IncarnationID.String(), "generation", selected.Generation)
230280 }
231281
232- if err := d.restoreSession(ctx, selectedSnapshot, selectedGeneration.Generation, selected); err != nil {
282+ if err := d.restoreSession(ctx, selectedSnapshot, selectedGeneration.Generation, selected, epoch); err != nil {
233283 return err
234284 }
235285 d.setStoppedRecovery(record, protocol.SessionDown)
M internal/usecase/daemon/snapshot_scheduler_test.go +92 -0
@@ -83,6 +83,98 @@ func TestSnapshotCoordinatorQuarantineJoinsInFlightPublication(t *testing.T) {
8383 }
8484 }
8585
86+// TestQuarantineDiscardDoesNotDoubleDecrementPendingCaptures proves the pending
87+// capture accounting is exact across a quarantine discard and a rollback
88+// replacement: a queued capture the quarantine discarded already dropped its own
89+// pending count, so the worker-side finish that rejects its stale identity must
90+// not decrement a replacement scheduled after the rollback.
91+func TestQuarantineDiscardDoesNotDoubleDecrementPendingCaptures(t *testing.T) {
92+ d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), stubClock{})
93+ repository := portsmocks.NewMockSnapshotRepository(t)
94+ WithSnapshotRepository(repository)(d)
95+ startSnapshotEncodeWorker(t, d)
96+
97+ blockerStarted := make(chan struct{})
98+ releaseBlocker := make(chan struct{})
99+ var releaseOnce sync.Once
100+ release := func() { releaseOnce.Do(func() { close(releaseBlocker) }) }
101+ t.Cleanup(release)
102+ repository.EXPECT().Publish(mock.Anything, mock.Anything).RunAndReturn(func(_ context.Context, publication ports.SnapshotPublication) error {
103+ if publication.Name == "blocker" {
104+ close(blockerStarted)
105+ <-releaseBlocker
106+ }
107+ return nil
108+ }).Maybe()
109+
110+ // Occupy the single worker so the test session's capture stays queued.
111+ blocker := newSnapshotTestSession(t, "blocker", false, "/work")
112+ markSnapshotDirty(blocker)
113+ require.True(t, d.scheduleSnapshot(blocker))
114+ awaitTestValue(t, blockerStarted, "the blocking publication did not start")
115+
116+ sess := newSnapshotTestSession(t, "work", false, "/work")
117+ markSnapshotDirty(sess)
118+ require.True(t, d.scheduleSnapshot(sess), "the test capture should queue behind the blocked worker")
119+ sess.snapshotMu.Lock()
120+ first := sess.snapshotQueuedCapture
121+ firstCount := sess.snapshotPendingCaptures
122+ sess.snapshotMu.Unlock()
123+ require.NotNil(t, first)
124+ require.Equal(t, uint(1), firstCount)
125+
126+ // Quarantine discards the queued capture and drops its pending count exactly
127+ // once.
128+ <-quarantineSnapshotCoordinator(sess)
129+ sess.snapshotMu.Lock()
130+ discardedCount := sess.snapshotPendingCaptures
131+ sess.snapshotMu.Unlock()
132+ require.Equal(t, uint(0), discardedCount, "quarantine must consume the discarded capture's count")
133+ require.True(t, first.coordinatorDiscarded)
134+
135+ // Model the worker having already dequeued the stale capture: the global
136+ // bounded queue still holds it, so free the slot before the replacement.
137+ select {
138+ case dequeued := <-d.snapshotJobs:
139+ require.Same(t, first, dequeued)
140+ default:
141+ t.Fatal("the discarded capture must still be queued in the global worker queue")
142+ }
143+
144+ // The rollback restores scheduling, and the replacement capture owns its own
145+ // pending count.
146+ rollbackSnapshotQuarantine(sess, true)
147+ require.True(t, d.scheduleSnapshot(sess), "the rolled-back session must schedule a replacement capture")
148+ sess.snapshotMu.Lock()
149+ replacement := sess.snapshotQueuedCapture
150+ replacementCount := sess.snapshotPendingCaptures
151+ sess.snapshotMu.Unlock()
152+ require.NotNil(t, replacement)
153+ require.NotSame(t, first, replacement)
154+ require.Equal(t, uint(1), replacementCount)
155+
156+ // The worker finishing the stale discarded capture must not decrement the
157+ // live replacement's pending count.
158+ d.finishSnapshotCapture(first, false)
159+ sess.snapshotMu.Lock()
160+ afterFinishCount := sess.snapshotPendingCaptures
161+ afterFinishPending := sess.snapshotPending
162+ sess.snapshotMu.Unlock()
163+ require.Equal(t, uint(1), afterFinishCount, "the discarded capture must not decrement the replacement")
164+ require.True(t, afterFinishPending)
165+
166+ // Once the worker drains both captures only the replacement publishes, and
167+ // the accounting returns to zero.
168+ release()
169+ awaitSnapshotClean(t, sess)
170+ sess.snapshotMu.Lock()
171+ finalCount := sess.snapshotPendingCaptures
172+ finalPending := sess.snapshotPending
173+ sess.snapshotMu.Unlock()
174+ require.Equal(t, uint(0), finalCount)
175+ require.False(t, finalPending)
176+}
177+
86178 func TestForcedSnapshotSchedulesSuccessorAfterRoutineInFlight(t *testing.T) {
87179 d := newTestDaemon(t, portsmocks.NewMockPTYFactory(t), stubClock{})
88180 repository := portsmocks.NewMockSnapshotRepository(t)

Keyboard shortcuts

Single-key shortcuts are on. ctrl k and esc work whatever this says.

ctrl k :
go to a section, branch, tag, file or repository
?
this list
j k
move the row cursor down / up
enter
open the row under the cursor; on Files, a directory opens or closes in place
↑ ↓ ← → home end
in the file tree (focus on a row): previous / next row, close / open a directory or go to its parent / first entry, first / last row. They never open a file
l h
on Files: open a directory or a file / close a directory or go up to its parent, then to the parent page. Elsewhere: into the side panel (on a merge request, its files) / back to the content; there j k move between its controls
g g G
first / last row
/
filter the list; on Files, find a file anywhere in the repository (esc clears, enter opens the best match)
b
switch branch or tag
g s g f g c g b g m g i g r g a
tabs: summary, files, commits, branches & tags, merge requests, issues, releases, actions (when available)
g h
home
w
wide mode: use the whole width of the screen, or not (remembered)
y
copy a permalink pinned to the commit
d u x
diff: split / unified / collapse or expand all
e E
diff: the whole file in view / every file, with the lines the hunks hide
n
merge request: next note in the diff
esc
close a dialog, leave a field

Branches 12

Tags 11

↑↓ moveenter openesc close