feat(subagents): queue follow-ups and reset run budgets
- commit
- cc1869fe38c951f10c0035f8b99f9830a916bd69
- author
- brice · · 2026-08-08 07:09 UTC
- parent
- ab30a57
- signature
- GPG Verified
CD88F9E564E0102646FB6077A1A07C7C7C3E349D· arch-gpg Verified for the committer email
22 files · +1442 -185
M docs/subagents.md +3 -3
| @@ -125,10 +125,10 @@ subagent://<id>/patch | |||
| 125 | 125 | ||
| 126 | 126 | Use `/subagents resume-session <id>` to continue the existing session without replaying its original task. Use `/subagents restart-task <id>` only when intentionally starting the stored task again. Cancellation requests graceful shutdown first, then forcefully cancels after the configured grace period. | |
| 127 | 127 | ||
| 128 | - | When auto-subagents is enabled and permitted by launch-time policy, the model can manage workers with `subagent_status`, `subagent_stop`, and `subagent_resume`. Call `subagent_status` with `{}` to list workers visible to the active supervisor session, or pass `{"agent_id":"<id-or-unique-prefix>"}` to query one worker. Queries use in-process snapshots and never wait for a worker turn or process to finish; repeated calls or state inspection are not completion signals. The JSON response contains the worker id, a normalized lifecycle state (`starting`, `running`, `completed`, `failed`, `cancelled`, or `detached`), the underlying `process_state` and `turn_state`, start/update/finish timestamps, a bounded first-line task summary, and terminal-result metadata/reference when available. It intentionally omits prompts, transcripts, result output, credentials, provider settings, and filesystem paths. | |
| 128 | + | When auto-subagents is enabled and permitted by launch-time policy, the model can manage workers with `subagent_status`, `subagent_stop`, and `subagent_resume`. Call `subagent_status` with `{}` to list workers visible to the active supervisor session, or pass `{"agent_id":"<id-or-unique-prefix>"}` to query one worker. Queries use in-process snapshots and never wait for a worker turn or process to finish; repeated calls or state inspection are not completion signals. The JSON response contains the worker id, a normalized lifecycle state (`starting`, `running`, `completed`, `failed`, `cancelled`, or `detached`), the underlying `process_state` and `turn_state`, `lifetime_turns` and `current_run_turns` counters, start/update/finish timestamps, a bounded first-line task summary, and terminal-result metadata/reference when available. It intentionally omits prompts, transcripts, result output, credentials, provider settings, and filesystem paths. | |
| 129 | 129 | ||
| 130 | - | Use `subagent_stop` with an `agent_id` to request termination of a stuck worker through the normal graceful-then-forceful supervisor lifecycle. Its `stop_requested` result confirms that shutdown has started; wait for the host completion update before attempting a restart. Use `subagent_resume` with an `agent_id` and `prompt` to send a new follow-up to an idle worker or to restart a terminal worker from its retained session with that follow-up as the initial turn. It never replays the original task, and its completion is delivered to the manager through the normal auto-subagent update. An unknown or ambiguous id is returned as a model-visible tool error; the existing `/subagents` dashboard and final-result notifications remain unchanged. An explicit `--tools` allowlist must include each manager tool that the model needs, and `subagent_status` must not be polled solely to wait for completion. | |
| 130 | + | Use `subagent_stop` with an `agent_id` to request termination of a stuck worker through the normal graceful-then-forceful supervisor lifecycle. Its `stop_requested` result confirms that shutdown has started; wait for the host completion update before attempting a restart. Use `subagent_resume` with an `agent_id` and `prompt` to start a fresh max-turn budget while retaining the transcript/session/context, send the follow-up to an idle worker, queue it for the next available message turn when the worker is active, or restart a terminal worker from its retained session with that follow-up as the initial turn. A follow-up remains durably pending until its matching user message is flushed to the session and the delegated `turn.started` boundary is recorded; a worker-ready event alone does not acknowledge it. Automatic retries, model loops, and compaction remain within the current run budget; multiple active follow-ups are delivered in order. Commands have stable identities so cancellation or an ambiguous socket-write outcome can be retried without replaying an accepted command. It never replays the original task, and its completion is delivered to the manager through the normal auto-subagent update. An unknown or ambiguous id is returned as a model-visible tool error; the existing `/subagents` dashboard and final-result notifications remain unchanged. An explicit `--tools` allowlist must include each manager tool that the model needs, and `subagent_status` must not be polled solely to wait for completion. | |
| 131 | 131 | ||
| 132 | 132 | ## Resource policy | |
| 133 | 133 | ||
| 134 | - | The persisted `subagents` config object supports `max_concurrent`, `max_concurrent_per_parent`, `queue_timeout`, `default_timeout`, `max_turns`, output caps, allowed tools/roots, heartbeat and idle timeouts. Previously saved `max_total_spawned` values are accepted but ignored. A missing or non-positive configured `max_turns` uses the default ceiling of 3. The `max_turns` field in `subagent_spawn` is optional: omit it to use the policy ceiling, or provide a value from 1 through the configured maximum. Concurrency limits apply to slash commands, `subagent_spawn`, and batch operations. A child cannot create descendants in v1. Per-agent timeouts are retained across reload/resume. Packaged `zut run` agents keep their declared capability ceiling by disabling subagent delegation; profiles are not a substitute for that permission boundary. A primary whose launch-time policy disables delegation still receives the strict orchestrator contract, but does not receive actionable profile metadata or a replacement implementation path. | |
| 134 | + | The persisted `subagents` config object supports `max_concurrent`, `max_concurrent_per_parent`, `queue_timeout`, `default_timeout`, `max_turns`, output caps, allowed tools/roots, heartbeat and idle timeouts. `max_turns` bounds one active worker run; lifetime and current-run turn counters remain persisted and visible in worker status. An explicit `subagent_resume` starts a new run with a fresh budget without replaying the original task or losing session context. Previously saved `max_total_spawned` values are accepted but ignored. A missing or non-positive configured `max_turns` uses the default ceiling of 3. The `max_turns` field in `subagent_spawn` is optional: omit it to use the policy ceiling, or provide a value from 1 through the configured maximum. Concurrency limits apply to slash commands, `subagent_spawn`, and batch operations. A child cannot create descendants in v1. Per-agent timeouts are retained across reload/resume. Packaged `zut run` agents keep their declared capability ceiling by disabling subagent delegation; profiles are not a substitute for that permission boundary. A primary whose launch-time policy disables delegation still receives the strict orchestrator contract, but does not receive actionable profile metadata or a replacement implementation path. | |
| @@ -125,10 +125,10 @@ subagent://<id>/patch | |||||
| 125 | 125 | ||||
| 126 | Use `/subagents resume-session <id>` to continue the existing session without replaying its original task. Use `/subagents restart-task <id>` only when intentionally starting the stored task again. Cancellation requests graceful shutdown first, then forcefully cancels after the configured grace period. | 126 | Use `/subagents resume-session <id>` to continue the existing session without replaying its original task. Use `/subagents restart-task <id>` only when intentionally starting the stored task again. Cancellation requests graceful shutdown first, then forcefully cancels after the configured grace period. | ||
| 127 | 127 | ||||
| 128 | - | When auto-subagents is enabled and permitted by launch-time policy, the model can manage workers with `subagent_status`, `subagent_stop`, and `subagent_resume`. Call `subagent_status` with `{}` to list workers visible to the active supervisor session, or pass `{"agent_id":"<id-or-unique-prefix>"}` to query one worker. Queries use in-process snapshots and never wait for a worker turn or process to finish; repeated calls or state inspection are not completion signals. The JSON response contains the worker id, a normalized lifecycle state (`starting`, `running`, `completed`, `failed`, `cancelled`, or `detached`), the underlying `process_state` and `turn_state`, start/update/finish timestamps, a bounded first-line task summary, and terminal-result metadata/reference when available. It intentionally omits prompts, transcripts, result output, credentials, provider settings, and filesystem paths. | 128 | + | When auto-subagents is enabled and permitted by launch-time policy, the model can manage workers with `subagent_status`, `subagent_stop`, and `subagent_resume`. Call `subagent_status` with `{}` to list workers visible to the active supervisor session, or pass `{"agent_id":"<id-or-unique-prefix>"}` to query one worker. Queries use in-process snapshots and never wait for a worker turn or process to finish; repeated calls or state inspection are not completion signals. The JSON response contains the worker id, a normalized lifecycle state (`starting`, `running`, `completed`, `failed`, `cancelled`, or `detached`), the underlying `process_state` and `turn_state`, `lifetime_turns` and `current_run_turns` counters, start/update/finish timestamps, a bounded first-line task summary, and terminal-result metadata/reference when available. It intentionally omits prompts, transcripts, result output, credentials, provider settings, and filesystem paths. |
| 129 | 129 | ||||
| 130 | - | Use `subagent_stop` with an `agent_id` to request termination of a stuck worker through the normal graceful-then-forceful supervisor lifecycle. Its `stop_requested` result confirms that shutdown has started; wait for the host completion update before attempting a restart. Use `subagent_resume` with an `agent_id` and `prompt` to send a new follow-up to an idle worker or to restart a terminal worker from its retained session with that follow-up as the initial turn. It never replays the original task, and its completion is delivered to the manager through the normal auto-subagent update. An unknown or ambiguous id is returned as a model-visible tool error; the existing `/subagents` dashboard and final-result notifications remain unchanged. An explicit `--tools` allowlist must include each manager tool that the model needs, and `subagent_status` must not be polled solely to wait for completion. | 130 | + | Use `subagent_stop` with an `agent_id` to request termination of a stuck worker through the normal graceful-then-forceful supervisor lifecycle. Its `stop_requested` result confirms that shutdown has started; wait for the host completion update before attempting a restart. Use `subagent_resume` with an `agent_id` and `prompt` to start a fresh max-turn budget while retaining the transcript/session/context, send the follow-up to an idle worker, queue it for the next available message turn when the worker is active, or restart a terminal worker from its retained session with that follow-up as the initial turn. A follow-up remains durably pending until its matching user message is flushed to the session and the delegated `turn.started` boundary is recorded; a worker-ready event alone does not acknowledge it. Automatic retries, model loops, and compaction remain within the current run budget; multiple active follow-ups are delivered in order. Commands have stable identities so cancellation or an ambiguous socket-write outcome can be retried without replaying an accepted command. It never replays the original task, and its completion is delivered to the manager through the normal auto-subagent update. An unknown or ambiguous id is returned as a model-visible tool error; the existing `/subagents` dashboard and final-result notifications remain unchanged. An explicit `--tools` allowlist must include each manager tool that the model needs, and `subagent_status` must not be polled solely to wait for completion. |
| 131 | 131 | ||||
| 132 | ## Resource policy | 132 | ## Resource policy | ||
| 133 | 133 | ||||
| 134 | - | The persisted `subagents` config object supports `max_concurrent`, `max_concurrent_per_parent`, `queue_timeout`, `default_timeout`, `max_turns`, output caps, allowed tools/roots, heartbeat and idle timeouts. Previously saved `max_total_spawned` values are accepted but ignored. A missing or non-positive configured `max_turns` uses the default ceiling of 3. The `max_turns` field in `subagent_spawn` is optional: omit it to use the policy ceiling, or provide a value from 1 through the configured maximum. Concurrency limits apply to slash commands, `subagent_spawn`, and batch operations. A child cannot create descendants in v1. Per-agent timeouts are retained across reload/resume. Packaged `zut run` agents keep their declared capability ceiling by disabling subagent delegation; profiles are not a substitute for that permission boundary. A primary whose launch-time policy disables delegation still receives the strict orchestrator contract, but does not receive actionable profile metadata or a replacement implementation path. | 134 | + | The persisted `subagents` config object supports `max_concurrent`, `max_concurrent_per_parent`, `queue_timeout`, `default_timeout`, `max_turns`, output caps, allowed tools/roots, heartbeat and idle timeouts. `max_turns` bounds one active worker run; lifetime and current-run turn counters remain persisted and visible in worker status. An explicit `subagent_resume` starts a new run with a fresh budget without replaying the original task or losing session context. Previously saved `max_total_spawned` values are accepted but ignored. A missing or non-positive configured `max_turns` uses the default ceiling of 3. The `max_turns` field in `subagent_spawn` is optional: omit it to use the policy ceiling, or provide a value from 1 through the configured maximum. Concurrency limits apply to slash commands, `subagent_spawn`, and batch operations. A child cannot create descendants in v1. Per-agent timeouts are retained across reload/resume. Packaged `zut run` agents keep their declared capability ceiling by disabling subagent delegation; profiles are not a substitute for that permission boundary. A primary whose launch-time policy disables delegation still receives the strict orchestrator contract, but does not receive actionable profile metadata or a replacement implementation path. |
M packages/agent/args.go +26 -1
| @@ -139,8 +139,13 @@ type Args struct { | |||
| 139 | 139 | // --subagent-worker. | |
| 140 | 140 | SubagentWorker string | |
| 141 | 141 | ||
| 142 | - | // SubagentMaxTurns limits prompt-level turns in worker mode. | |
| 142 | + | // SubagentMaxTurns limits message turns in one worker run. | |
| 143 | 143 | SubagentMaxTurns int | |
| 144 | + | // SubagentLifetimeTurns and SubagentRunTurns are supervisor-provided | |
| 145 | + | // counters used to resume a worker without losing observability or its | |
| 146 | + | // current-run budget. They are internal worker flags, not user policy. | |
| 147 | + | SubagentLifetimeTurns int | |
| 148 | + | SubagentRunTurns int | |
| 144 | 149 | ||
| 145 | 150 | // Subagent selects a named markdown profile for a subagent child. | |
| 146 | 151 | // It is intentionally an internal child flag; the parent subagent tool | |
| @@ -371,6 +376,26 @@ func ParseArgs(in []string) (Args, error) { | |||
| 371 | 376 | return a, fmt.Errorf("--max-turns must be a positive integer") | |
| 372 | 377 | } | |
| 373 | 378 | a.SubagentMaxTurns = n | |
| 379 | + | case "--subagent-lifetime-turns": | |
| 380 | + | v, err := want(&i, arg) | |
| 381 | + | if err != nil { | |
| 382 | + | return a, err | |
| 383 | + | } | |
| 384 | + | var n int | |
| 385 | + | if _, err := fmt.Sscanf(v, "%d", &n); err != nil || n < 0 { | |
| 386 | + | return a, fmt.Errorf("--subagent-lifetime-turns must be a non-negative integer") | |
| 387 | + | } | |
| 388 | + | a.SubagentLifetimeTurns = n | |
| 389 | + | case "--subagent-run-turns": | |
| 390 | + | v, err := want(&i, arg) | |
| 391 | + | if err != nil { | |
| 392 | + | return a, err | |
| 393 | + | } | |
| 394 | + | var n int | |
| 395 | + | if _, err := fmt.Sscanf(v, "%d", &n); err != nil || n < 0 { | |
| 396 | + | return a, fmt.Errorf("--subagent-run-turns must be a non-negative integer") | |
| 397 | + | } | |
| 398 | + | a.SubagentRunTurns = n | |
| 374 | 399 | default: | |
| 375 | 400 | if strings.HasPrefix(arg, "-") && arg != "-" { | |
| 376 | 401 | return a, fmt.Errorf("unknown flag %q", arg) | |
| @@ -139,8 +139,13 @@ type Args struct { | |||||
| 139 | // --subagent-worker. | 139 | // --subagent-worker. | ||
| 140 | SubagentWorker string | 140 | SubagentWorker string | ||
| 141 | 141 | ||||
| 142 | - | // SubagentMaxTurns limits prompt-level turns in worker mode. | 142 | + | // SubagentMaxTurns limits message turns in one worker run. |
| 143 | SubagentMaxTurns int | 143 | SubagentMaxTurns int | ||
| 144 | + | // SubagentLifetimeTurns and SubagentRunTurns are supervisor-provided | |||
| 145 | + | // counters used to resume a worker without losing observability or its | |||
| 146 | + | // current-run budget. They are internal worker flags, not user policy. | |||
| 147 | + | SubagentLifetimeTurns int | |||
| 148 | + | SubagentRunTurns int | |||
| 144 | 149 | ||||
| 145 | // Subagent selects a named markdown profile for a subagent child. | 150 | // Subagent selects a named markdown profile for a subagent child. | ||
| 146 | // It is intentionally an internal child flag; the parent subagent tool | 151 | // It is intentionally an internal child flag; the parent subagent tool | ||
| @@ -371,6 +376,26 @@ func ParseArgs(in []string) (Args, error) { | |||||
| 371 | return a, fmt.Errorf("--max-turns must be a positive integer") | 376 | return a, fmt.Errorf("--max-turns must be a positive integer") | ||
| 372 | } | 377 | } | ||
| 373 | a.SubagentMaxTurns = n | 378 | a.SubagentMaxTurns = n | ||
| 379 | + | case "--subagent-lifetime-turns": | |||
| 380 | + | v, err := want(&i, arg) | |||
| 381 | + | if err != nil { | |||
| 382 | + | return a, err | |||
| 383 | + | } | |||
| 384 | + | var n int | |||
| 385 | + | if _, err := fmt.Sscanf(v, "%d", &n); err != nil || n < 0 { | |||
| 386 | + | return a, fmt.Errorf("--subagent-lifetime-turns must be a non-negative integer") | |||
| 387 | + | } | |||
| 388 | + | a.SubagentLifetimeTurns = n | |||
| 389 | + | case "--subagent-run-turns": | |||
| 390 | + | v, err := want(&i, arg) | |||
| 391 | + | if err != nil { | |||
| 392 | + | return a, err | |||
| 393 | + | } | |||
| 394 | + | var n int | |||
| 395 | + | if _, err := fmt.Sscanf(v, "%d", &n); err != nil || n < 0 { | |||
| 396 | + | return a, fmt.Errorf("--subagent-run-turns must be a non-negative integer") | |||
| 397 | + | } | |||
| 398 | + | a.SubagentRunTurns = n | |||
| 374 | default: | 399 | default: | ||
| 375 | if strings.HasPrefix(arg, "-") && arg != "-" { | 400 | if strings.HasPrefix(arg, "-") && arg != "-" { | ||
| 376 | return a, fmt.Errorf("unknown flag %q", arg) | 401 | return a, fmt.Errorf("unknown flag %q", arg) | ||
M packages/agent/subagent_worker.go +142 -44
| @@ -36,6 +36,56 @@ import ( | |||
| 36 | 36 | // production; tests use the stubchild binary under | |
| 37 | 37 | // packages/agent/subagents/testdata/cmd/stubchild instead of the real model | |
| 38 | 38 | // loop. | |
| 39 | + | // workerTurnBudget is deliberately independent of the model loop: one | |
| 40 | + | // admitted message turn consumes one current-run and lifetime turn, while | |
| 41 | + | // retries, compaction, and provider/tool loops remain inside that turn. | |
| 42 | + | type workerTurnBudget struct { | |
| 43 | + | sequence int | |
| 44 | + | lifetime int | |
| 45 | + | current int | |
| 46 | + | } | |
| 47 | + | ||
| 48 | + | type sessionPersistenceState struct { | |
| 49 | + | mu sync.Mutex | |
| 50 | + | errValue error | |
| 51 | + | } | |
| 52 | + | ||
| 53 | + | func (s *sessionPersistenceState) record(err error) { | |
| 54 | + | if err == nil { | |
| 55 | + | return | |
| 56 | + | } | |
| 57 | + | s.mu.Lock() | |
| 58 | + | if s.errValue == nil { | |
| 59 | + | s.errValue = err | |
| 60 | + | } | |
| 61 | + | s.mu.Unlock() | |
| 62 | + | } | |
| 63 | + | ||
| 64 | + | func (s *sessionPersistenceState) failed() bool { | |
| 65 | + | s.mu.Lock() | |
| 66 | + | defer s.mu.Unlock() | |
| 67 | + | return s.errValue != nil | |
| 68 | + | } | |
| 69 | + | ||
| 70 | + | func (s *sessionPersistenceState) err() error { | |
| 71 | + | s.mu.Lock() | |
| 72 | + | defer s.mu.Unlock() | |
| 73 | + | return s.errValue | |
| 74 | + | } | |
| 75 | + | ||
| 76 | + | func (b *workerTurnBudget) start(maxTurns int, newRun bool) (step, lifetime, current int, admitted bool) { | |
| 77 | + | if newRun { | |
| 78 | + | b.current = 0 | |
| 79 | + | } | |
| 80 | + | b.sequence++ | |
| 81 | + | if maxTurns > 0 && b.current >= maxTurns { | |
| 82 | + | return b.sequence, b.lifetime, b.current, false | |
| 83 | + | } | |
| 84 | + | b.lifetime++ | |
| 85 | + | b.current++ | |
| 86 | + | return b.sequence, b.lifetime, b.current, true | |
| 87 | + | } | |
| 88 | + | ||
| 39 | 89 | func runSubagentWorkerMode(ctx context.Context, args Args, version string) error { | |
| 40 | 90 | if args.SubagentWorker == "" { | |
| 41 | 91 | return fmt.Errorf("--subagent-worker requires a socket path") | |
| @@ -74,6 +124,7 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) error | |||
| 74 | 124 | if err != nil { | |
| 75 | 125 | return err | |
| 76 | 126 | } | |
| 127 | + | var sessionPersistence sessionPersistenceState | |
| 77 | 128 | if sess != nil { | |
| 78 | 129 | var providerName, model string | |
| 79 | 130 | sess, ag, providerName, model, err = applyInitialSessionResume(ctx, args, r, extMgr, sess, ag) | |
| @@ -82,6 +133,18 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) error | |||
| 82 | 133 | } | |
| 83 | 134 | r.Provider, r.Model = providerName, model | |
| 84 | 135 | ag.OnToolResult = func(_ string, result core.ToolResult) { persistExtensionToolResult(extMgr, sess, result) } | |
| 136 | + | // A prompt is acknowledged by the supervisor only after this callback | |
| 137 | + | // has appended and synced the matching user message. This closes the | |
| 138 | + | // crash window between accepting a follow-up and writing session.json. | |
| 139 | + | ag.OnMessageAppended = func(message provider.Message) { | |
| 140 | + | if err := sess.AppendMessage(message); err != nil { | |
| 141 | + | sessionPersistence.record(err) | |
| 142 | + | return | |
| 143 | + | } | |
| 144 | + | if err := sess.Sync(); err != nil { | |
| 145 | + | sessionPersistence.record(err) | |
| 146 | + | } | |
| 147 | + | } | |
| 85 | 148 | defer sess.Close() | |
| 86 | 149 | } | |
| 87 | 150 | announceSession(extMgr, sess) | |
| @@ -113,9 +176,12 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) error | |||
| 113 | 176 | em := newSubagentEmitter(os.Stdout, logMirror) | |
| 114 | 177 | em.setProtocolIdentity(os.Getenv("ZUT_SUBAGENT_AGENT_ID")) | |
| 115 | 178 | em.emit("agent.ready", map[string]any{ | |
| 116 | - | "version": version, | |
| 117 | - | "cwd": r.CWD, | |
| 118 | - | "model": r.Model, | |
| 179 | + | "version": version, | |
| 180 | + | "cwd": r.CWD, | |
| 181 | + | "model": r.Model, | |
| 182 | + | "lifetime_turns": args.SubagentLifetimeTurns, | |
| 183 | + | "current_run_turns": args.SubagentRunTurns, | |
| 184 | + | "max_turns": args.SubagentMaxTurns, | |
| 119 | 185 | }) | |
| 120 | 186 | ||
| 121 | 187 | // Heartbeats make a live-but-detached child distinguishable from a | |
| @@ -145,63 +211,75 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) error | |||
| 145 | 211 | // Keep a per-turn cancel so the "cancel" inbox message can interrupt | |
| 146 | 212 | // an in-flight turn without tearing down the whole daemon. | |
| 147 | 213 | var ( | |
| 148 | - | mu sync.Mutex | |
| 149 | - | cancelFn context.CancelFunc | |
| 150 | - | busyTurn bool | |
| 151 | - | turnNo = initialTurnNumber(os.Getenv("ZUT_SUBAGENT_EVENT_LOG")) | |
| 214 | + | mu sync.Mutex | |
| 215 | + | cancelFn context.CancelFunc | |
| 216 | + | busyTurn bool | |
| 217 | + | budget = workerTurnBudget{ | |
| 218 | + | sequence: initialTurnNumber(os.Getenv("ZUT_SUBAGENT_EVENT_LOG")), | |
| 219 | + | lifetime: args.SubagentLifetimeTurns, | |
| 220 | + | current: args.SubagentRunTurns, | |
| 221 | + | } | |
| 152 | 222 | turnWG sync.WaitGroup | |
| 153 | 223 | closing bool | |
| 224 | + | turnPending bool | |
| 225 | + | seenCommands = make(map[string]struct{}) | |
| 154 | 226 | shutdown = make(chan struct{}) | |
| 155 | 227 | shutdownOnce sync.Once | |
| 156 | 228 | ) | |
| 157 | 229 | maxOutputBytes, maxOutputLines := workerOutputLimits() | |
| 158 | 230 | ||
| 159 | - | runOne := func(prompt string) { | |
| 231 | + | runOne := func(prompt string, newRun bool, commandID string) { | |
| 160 | 232 | mu.Lock() | |
| 161 | 233 | // launchTurn admits the goroutine to turnWG before it starts so | |
| 162 | 234 | // shutdown can join it. A shutdown can win the scheduling race | |
| 163 | 235 | // after that admission but before this goroutine acquires mu; in | |
| 164 | 236 | // that case do not start a new Prompt after waitForActiveTurn has | |
| 165 | 237 | // declared the worker closing. | |
| 166 | - | if closing { | |
| 238 | + | if closing || busyTurn { | |
| 167 | 239 | mu.Unlock() | |
| 240 | + | em.emit("error", map[string]any{"code": "turn_rejected", "reason": "busy", "command_id": commandID, "message": "worker is busy"}) | |
| 168 | 241 | return | |
| 169 | 242 | } | |
| 170 | - | if args.SubagentMaxTurns > 0 && turnNo >= args.SubagentMaxTurns { | |
| 171 | - | // Consume a unique rejected attempt so repeated prompts after | |
| 172 | - | // the limit cannot reuse the same turn id in the event log. | |
| 173 | - | turnNo++ | |
| 174 | - | step := turnNo | |
| 243 | + | busyTurn = true | |
| 244 | + | turnPending = false | |
| 245 | + | step, lifetime, currentRun, admitted := budget.start(args.SubagentMaxTurns, newRun) | |
| 246 | + | if !admitted { | |
| 247 | + | busyTurn = false | |
| 175 | 248 | mu.Unlock() | |
| 176 | 249 | turnID := fmt.Sprintf("turn-%d", step) | |
| 177 | - | errPayload := map[string]any{"code": "max_turns", "message": "maximum subagent turns reached"} | |
| 250 | + | errPayload := map[string]any{"code": "turn_rejected", "reason": "max_turns", "command_id": commandID, "message": "maximum subagent turns reached"} | |
| 251 | + | em.emit("error", errPayload) | |
| 178 | 252 | em.emit("turn.result", map[string]any{"status": "failed", "turn_id": turnID, "error": errPayload}) | |
| 179 | 253 | em.emit("turn.failed", map[string]any{"turn_id": turnID, "error": errPayload}) | |
| 180 | 254 | em.emit("turn_end", map[string]any{"step": step, "turn_id": turnID, "error": errPayload["message"]}) | |
| 181 | - | em.emit("agent.idle", map[string]any{"turn_id": turnID}) | |
| 182 | - | return | |
| 183 | - | } | |
| 184 | - | if busyTurn { | |
| 185 | - | // Drop concurrent turns rather than queuing. The | |
| 186 | - | // supervisor protocol assumes one outstanding turn per | |
| 187 | - | // agent; if a user really wants to interrupt and start | |
| 188 | - | // another, they should send "cancel" first. | |
| 189 | - | mu.Unlock() | |
| 190 | - | em.emit("error", map[string]any{"message": "agent busy; send 'cancel' first"}) | |
| 191 | 255 | return | |
| 192 | 256 | } | |
| 193 | - | busyTurn = true | |
| 194 | - | turnNo++ | |
| 195 | - | step := turnNo | |
| 257 | + | turnID := fmt.Sprintf("turn-%d", step) | |
| 196 | 258 | c, cancel := context.WithCancel(ctx) | |
| 197 | 259 | cancelFn = cancel | |
| 198 | 260 | mu.Unlock() | |
| 199 | 261 | ||
| 200 | - | turnID := fmt.Sprintf("turn-%d", step) | |
| 201 | - | em.emit("turn.started", map[string]any{"step": step, "turn_id": turnID}) | |
| 202 | - | ||
| 262 | + | turnStarted := false | |
| 203 | 263 | sink := func(ev core.AgentEvent) { | |
| 204 | - | em.emit(ev.Type(), subagentEventData(ev)) | |
| 264 | + | data := subagentEventData(ev) | |
| 265 | + | if ev.Type() == "user_message" && !turnStarted { | |
| 266 | + | // Prompt invokes OnMessageAppended before delivering this sink | |
| 267 | + | // event, so the session row is durable before the lifecycle | |
| 268 | + | // acknowledgement is written. | |
| 269 | + | em.emit(ev.Type(), data) | |
| 270 | + | if !sessionPersistence.failed() { | |
| 271 | + | em.emit("turn.started", map[string]any{"step": step, "turn_id": turnID, "command_id": commandID, "lifetime_turns": lifetime, "current_run_turns": currentRun}) | |
| 272 | + | turnStarted = true | |
| 273 | + | } | |
| 274 | + | return | |
| 275 | + | } | |
| 276 | + | if ev.Type() == "turn_start" { | |
| 277 | + | // A provider/model-loop turn is nested inside the one delegated | |
| 278 | + | // message turn admitted above. Mark it on the wire so the | |
| 279 | + | // supervisor cannot mistake it for a new worker turn. | |
| 280 | + | data["nested_turn"] = true | |
| 281 | + | } | |
| 282 | + | em.emit(ev.Type(), data) | |
| 205 | 283 | } | |
| 206 | 284 | ||
| 207 | 285 | var persistCompaction func([]provider.Message) error | |
| @@ -209,7 +287,12 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) error | |||
| 209 | 287 | persistCompaction = sess.AppendCompaction | |
| 210 | 288 | } | |
| 211 | 289 | start, err := promptWithContextRecovery(c, ag, prompt, sink, persistCompaction) | |
| 212 | - | WriteNewTranscript(ag, sess, start) | |
| 290 | + | if sess != nil && ag.OnMessageAppended == nil { | |
| 291 | + | WriteNewTranscript(ag, sess, start) | |
| 292 | + | } | |
| 293 | + | if sessionErr := sessionPersistence.err(); err == nil && sessionErr != nil { | |
| 294 | + | err = sessionErr | |
| 295 | + | } | |
| 213 | 296 | ||
| 214 | 297 | output := finalAssistantText(ag.Messages()[start:]) | |
| 215 | 298 | resultStatus := "succeeded" | |
| @@ -245,24 +328,26 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) error | |||
| 245 | 328 | // The supervisor treats agent.idle as permission to start a follow-up. | |
| 246 | 329 | // Publish it only after the busy gate has opened, otherwise a manager | |
| 247 | 330 | // can send a turn that the inbox loop drops as still busy. | |
| 248 | - | em.emit("agent.idle", map[string]any{"turn_id": turnID}) | |
| 331 | + | em.emit("agent.idle", map[string]any{"turn_id": turnID, "lifetime_turns": lifetime, "current_run_turns": currentRun}) | |
| 249 | 332 | } | |
| 250 | 333 | ||
| 251 | - | launchTurn := func(prompt string) { | |
| 334 | + | launchTurn := func(prompt string, newRun bool, commandID string) bool { | |
| 252 | 335 | mu.Lock() | |
| 253 | - | if closing { | |
| 336 | + | if closing || busyTurn || turnPending { | |
| 254 | 337 | mu.Unlock() | |
| 255 | - | return | |
| 338 | + | em.emit("error", map[string]any{"code": "turn_rejected", "reason": "busy", "command_id": commandID, "message": "worker is busy"}) | |
| 339 | + | return false | |
| 256 | 340 | } | |
| 257 | - | // Register the goroutine before launching it. Shutdown can then | |
| 258 | - | // close the gate and wait for every turn that was admitted, even | |
| 259 | - | // if runOne has not acquired mu and set busyTurn yet. | |
| 341 | + | // Reserve the worker before launching the goroutine. A second command | |
| 342 | + | // cannot be accepted in the gap before runOne acquires mu. | |
| 343 | + | turnPending = true | |
| 260 | 344 | turnWG.Add(1) | |
| 261 | 345 | mu.Unlock() | |
| 262 | 346 | go func() { | |
| 263 | 347 | defer turnWG.Done() | |
| 264 | - | runOne(prompt) | |
| 348 | + | runOne(prompt, newRun, commandID) | |
| 265 | 349 | }() | |
| 350 | + | return true | |
| 266 | 351 | } | |
| 267 | 352 | ||
| 268 | 353 | waitForActiveTurn := func(cancel bool) { | |
| @@ -287,7 +372,7 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) error | |||
| 287 | 372 | // "starts working" the moment it boots, matching what users | |
| 288 | 373 | // expect from `/subagents new <task>`. | |
| 289 | 374 | if args.Prompt != "" { | |
| 290 | - | launchTurn(args.Prompt) | |
| 375 | + | launchTurn(args.Prompt, false, "") | |
| 291 | 376 | } | |
| 292 | 377 | ||
| 293 | 378 | // Inbox loop: one supervisor message at a time. We don't spawn | |
| @@ -333,10 +418,20 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) error | |||
| 333 | 418 | case subagents.CommandTurnStart: | |
| 334 | 419 | var payload subagents.TurnStartPayload | |
| 335 | 420 | if err := command.DecodePayload(&payload); err != nil { | |
| 336 | - | em.emit("error", map[string]any{"message": "invalid turn.start payload"}) | |
| 421 | + | em.emit("error", map[string]any{"code": "turn_rejected", "reason": "invalid_payload", "command_id": command.MessageID, "message": "invalid turn.start payload"}) | |
| 337 | 422 | continue | |
| 338 | 423 | } | |
| 339 | - | launchTurn(payload.Prompt) | |
| 424 | + | mu.Lock() | |
| 425 | + | _, duplicate := seenCommands[command.MessageID] | |
| 426 | + | mu.Unlock() | |
| 427 | + | if duplicate && command.MessageID != "" { | |
| 428 | + | continue | |
| 429 | + | } | |
| 430 | + | if launchTurn(payload.Prompt, payload.NewRun, command.MessageID) && command.MessageID != "" { | |
| 431 | + | mu.Lock() | |
| 432 | + | seenCommands[command.MessageID] = struct{}{} | |
| 433 | + | mu.Unlock() | |
| 434 | + | } | |
| 340 | 435 | default: | |
| 341 | 436 | // Unknown commands are rejected rather than interpreted as | |
| 342 | 437 | // child-originated control requests. | |
| @@ -555,6 +650,9 @@ func initialTurnNumber(path string) int { | |||
| 555 | 650 | } | |
| 556 | 651 | maxStep := 0 | |
| 557 | 652 | for _, ev := range events { | |
| 653 | + | if !subagents.IsDelegatedTurnStart(ev) { | |
| 654 | + | continue | |
| 655 | + | } | |
| 558 | 656 | if step, ok := ev.Data["step"].(float64); ok && int(step) > maxStep { | |
| 559 | 657 | maxStep = int(step) | |
| 560 | 658 | } | |
| @@ -36,6 +36,56 @@ import ( | |||||
| 36 | // production; tests use the stubchild binary under | 36 | // production; tests use the stubchild binary under | ||
| 37 | // packages/agent/subagents/testdata/cmd/stubchild instead of the real model | 37 | // packages/agent/subagents/testdata/cmd/stubchild instead of the real model | ||
| 38 | // loop. | 38 | // loop. | ||
| 39 | + | // workerTurnBudget is deliberately independent of the model loop: one | |||
| 40 | + | // admitted message turn consumes one current-run and lifetime turn, while | |||
| 41 | + | // retries, compaction, and provider/tool loops remain inside that turn. | |||
| 42 | + | type workerTurnBudget struct { | |||
| 43 | + | sequence int | |||
| 44 | + | lifetime int | |||
| 45 | + | current int | |||
| 46 | + | } | |||
| 47 | + | ||||
| 48 | + | type sessionPersistenceState struct { | |||
| 49 | + | mu sync.Mutex | |||
| 50 | + | errValue error | |||
| 51 | + | } | |||
| 52 | + | ||||
| 53 | + | func (s *sessionPersistenceState) record(err error) { | |||
| 54 | + | if err == nil { | |||
| 55 | + | return | |||
| 56 | + | } | |||
| 57 | + | s.mu.Lock() | |||
| 58 | + | if s.errValue == nil { | |||
| 59 | + | s.errValue = err | |||
| 60 | + | } | |||
| 61 | + | s.mu.Unlock() | |||
| 62 | + | } | |||
| 63 | + | ||||
| 64 | + | func (s *sessionPersistenceState) failed() bool { | |||
| 65 | + | s.mu.Lock() | |||
| 66 | + | defer s.mu.Unlock() | |||
| 67 | + | return s.errValue != nil | |||
| 68 | + | } | |||
| 69 | + | ||||
| 70 | + | func (s *sessionPersistenceState) err() error { | |||
| 71 | + | s.mu.Lock() | |||
| 72 | + | defer s.mu.Unlock() | |||
| 73 | + | return s.errValue | |||
| 74 | + | } | |||
| 75 | + | ||||
| 76 | + | func (b *workerTurnBudget) start(maxTurns int, newRun bool) (step, lifetime, current int, admitted bool) { | |||
| 77 | + | if newRun { | |||
| 78 | + | b.current = 0 | |||
| 79 | + | } | |||
| 80 | + | b.sequence++ | |||
| 81 | + | if maxTurns > 0 && b.current >= maxTurns { | |||
| 82 | + | return b.sequence, b.lifetime, b.current, false | |||
| 83 | + | } | |||
| 84 | + | b.lifetime++ | |||
| 85 | + | b.current++ | |||
| 86 | + | return b.sequence, b.lifetime, b.current, true | |||
| 87 | + | } | |||
| 88 | + | ||||
| 39 | func runSubagentWorkerMode(ctx context.Context, args Args, version string) error { | 89 | func runSubagentWorkerMode(ctx context.Context, args Args, version string) error { | ||
| 40 | if args.SubagentWorker == "" { | 90 | if args.SubagentWorker == "" { | ||
| 41 | return fmt.Errorf("--subagent-worker requires a socket path") | 91 | return fmt.Errorf("--subagent-worker requires a socket path") | ||
| @@ -74,6 +124,7 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) error | |||||
| 74 | if err != nil { | 124 | if err != nil { | ||
| 75 | return err | 125 | return err | ||
| 76 | } | 126 | } | ||
| 127 | + | var sessionPersistence sessionPersistenceState | |||
| 77 | if sess != nil { | 128 | if sess != nil { | ||
| 78 | var providerName, model string | 129 | var providerName, model string | ||
| 79 | sess, ag, providerName, model, err = applyInitialSessionResume(ctx, args, r, extMgr, sess, ag) | 130 | sess, ag, providerName, model, err = applyInitialSessionResume(ctx, args, r, extMgr, sess, ag) | ||
| @@ -82,6 +133,18 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) error | |||||
| 82 | } | 133 | } | ||
| 83 | r.Provider, r.Model = providerName, model | 134 | r.Provider, r.Model = providerName, model | ||
| 84 | ag.OnToolResult = func(_ string, result core.ToolResult) { persistExtensionToolResult(extMgr, sess, result) } | 135 | ag.OnToolResult = func(_ string, result core.ToolResult) { persistExtensionToolResult(extMgr, sess, result) } | ||
| 136 | + | // A prompt is acknowledged by the supervisor only after this callback | |||
| 137 | + | // has appended and synced the matching user message. This closes the | |||
| 138 | + | // crash window between accepting a follow-up and writing session.json. | |||
| 139 | + | ag.OnMessageAppended = func(message provider.Message) { | |||
| 140 | + | if err := sess.AppendMessage(message); err != nil { | |||
| 141 | + | sessionPersistence.record(err) | |||
| 142 | + | return | |||
| 143 | + | } | |||
| 144 | + | if err := sess.Sync(); err != nil { | |||
| 145 | + | sessionPersistence.record(err) | |||
| 146 | + | } | |||
| 147 | + | } | |||
| 85 | defer sess.Close() | 148 | defer sess.Close() | ||
| 86 | } | 149 | } | ||
| 87 | announceSession(extMgr, sess) | 150 | announceSession(extMgr, sess) | ||
| @@ -113,9 +176,12 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) error | |||||
| 113 | em := newSubagentEmitter(os.Stdout, logMirror) | 176 | em := newSubagentEmitter(os.Stdout, logMirror) | ||
| 114 | em.setProtocolIdentity(os.Getenv("ZUT_SUBAGENT_AGENT_ID")) | 177 | em.setProtocolIdentity(os.Getenv("ZUT_SUBAGENT_AGENT_ID")) | ||
| 115 | em.emit("agent.ready", map[string]any{ | 178 | em.emit("agent.ready", map[string]any{ | ||
| 116 | - | "version": version, | 179 | + | "version": version, |
| 117 | - | "cwd": r.CWD, | 180 | + | "cwd": r.CWD, |
| 118 | - | "model": r.Model, | 181 | + | "model": r.Model, |
| 182 | + | "lifetime_turns": args.SubagentLifetimeTurns, | |||
| 183 | + | "current_run_turns": args.SubagentRunTurns, | |||
| 184 | + | "max_turns": args.SubagentMaxTurns, | |||
| 119 | }) | 185 | }) | ||
| 120 | 186 | ||||
| 121 | // Heartbeats make a live-but-detached child distinguishable from a | 187 | // Heartbeats make a live-but-detached child distinguishable from a | ||
| @@ -145,63 +211,75 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) error | |||||
| 145 | // Keep a per-turn cancel so the "cancel" inbox message can interrupt | 211 | // Keep a per-turn cancel so the "cancel" inbox message can interrupt | ||
| 146 | // an in-flight turn without tearing down the whole daemon. | 212 | // an in-flight turn without tearing down the whole daemon. | ||
| 147 | var ( | 213 | var ( | ||
| 148 | - | mu sync.Mutex | 214 | + | mu sync.Mutex |
| 149 | - | cancelFn context.CancelFunc | 215 | + | cancelFn context.CancelFunc |
| 150 | - | busyTurn bool | 216 | + | busyTurn bool |
| 151 | - | turnNo = initialTurnNumber(os.Getenv("ZUT_SUBAGENT_EVENT_LOG")) | 217 | + | budget = workerTurnBudget{ |
| 218 | + | sequence: initialTurnNumber(os.Getenv("ZUT_SUBAGENT_EVENT_LOG")), | |||
| 219 | + | lifetime: args.SubagentLifetimeTurns, | |||
| 220 | + | current: args.SubagentRunTurns, | |||
| 221 | + | } | |||
| 152 | turnWG sync.WaitGroup | 222 | turnWG sync.WaitGroup | ||
| 153 | closing bool | 223 | closing bool | ||
| 224 | + | turnPending bool | |||
| 225 | + | seenCommands = make(map[string]struct{}) | |||
| 154 | shutdown = make(chan struct{}) | 226 | shutdown = make(chan struct{}) | ||
| 155 | shutdownOnce sync.Once | 227 | shutdownOnce sync.Once | ||
| 156 | ) | 228 | ) | ||
| 157 | maxOutputBytes, maxOutputLines := workerOutputLimits() | 229 | maxOutputBytes, maxOutputLines := workerOutputLimits() | ||
| 158 | 230 | ||||
| 159 | - | runOne := func(prompt string) { | 231 | + | runOne := func(prompt string, newRun bool, commandID string) { |
| 160 | mu.Lock() | 232 | mu.Lock() | ||
| 161 | // launchTurn admits the goroutine to turnWG before it starts so | 233 | // launchTurn admits the goroutine to turnWG before it starts so | ||
| 162 | // shutdown can join it. A shutdown can win the scheduling race | 234 | // shutdown can join it. A shutdown can win the scheduling race | ||
| 163 | // after that admission but before this goroutine acquires mu; in | 235 | // after that admission but before this goroutine acquires mu; in | ||
| 164 | // that case do not start a new Prompt after waitForActiveTurn has | 236 | // that case do not start a new Prompt after waitForActiveTurn has | ||
| 165 | // declared the worker closing. | 237 | // declared the worker closing. | ||
| 166 | - | if closing { | 238 | + | if closing || busyTurn { |
| 167 | mu.Unlock() | 239 | mu.Unlock() | ||
| 240 | + | em.emit("error", map[string]any{"code": "turn_rejected", "reason": "busy", "command_id": commandID, "message": "worker is busy"}) | |||
| 168 | return | 241 | return | ||
| 169 | } | 242 | } | ||
| 170 | - | if args.SubagentMaxTurns > 0 && turnNo >= args.SubagentMaxTurns { | 243 | + | busyTurn = true |
| 171 | - | // Consume a unique rejected attempt so repeated prompts after | 244 | + | turnPending = false |
| 172 | - | // the limit cannot reuse the same turn id in the event log. | 245 | + | step, lifetime, currentRun, admitted := budget.start(args.SubagentMaxTurns, newRun) |
| 173 | - | turnNo++ | 246 | + | if !admitted { |
| 174 | - | step := turnNo | 247 | + | busyTurn = false |
| 175 | mu.Unlock() | 248 | mu.Unlock() | ||
| 176 | turnID := fmt.Sprintf("turn-%d", step) | 249 | turnID := fmt.Sprintf("turn-%d", step) | ||
| 177 | - | errPayload := map[string]any{"code": "max_turns", "message": "maximum subagent turns reached"} | 250 | + | errPayload := map[string]any{"code": "turn_rejected", "reason": "max_turns", "command_id": commandID, "message": "maximum subagent turns reached"} |
| 251 | + | em.emit("error", errPayload) | |||
| 178 | em.emit("turn.result", map[string]any{"status": "failed", "turn_id": turnID, "error": errPayload}) | 252 | em.emit("turn.result", map[string]any{"status": "failed", "turn_id": turnID, "error": errPayload}) | ||
| 179 | em.emit("turn.failed", map[string]any{"turn_id": turnID, "error": errPayload}) | 253 | em.emit("turn.failed", map[string]any{"turn_id": turnID, "error": errPayload}) | ||
| 180 | em.emit("turn_end", map[string]any{"step": step, "turn_id": turnID, "error": errPayload["message"]}) | 254 | em.emit("turn_end", map[string]any{"step": step, "turn_id": turnID, "error": errPayload["message"]}) | ||
| 181 | - | em.emit("agent.idle", map[string]any{"turn_id": turnID}) | |||
| 182 | - | return | |||
| 183 | - | } | |||
| 184 | - | if busyTurn { | |||
| 185 | - | // Drop concurrent turns rather than queuing. The | |||
| 186 | - | // supervisor protocol assumes one outstanding turn per | |||
| 187 | - | // agent; if a user really wants to interrupt and start | |||
| 188 | - | // another, they should send "cancel" first. | |||
| 189 | - | mu.Unlock() | |||
| 190 | - | em.emit("error", map[string]any{"message": "agent busy; send 'cancel' first"}) | |||
| 191 | return | 255 | return | ||
| 192 | } | 256 | } | ||
| 193 | - | busyTurn = true | 257 | + | turnID := fmt.Sprintf("turn-%d", step) |
| 194 | - | turnNo++ | |||
| 195 | - | step := turnNo | |||
| 196 | c, cancel := context.WithCancel(ctx) | 258 | c, cancel := context.WithCancel(ctx) | ||
| 197 | cancelFn = cancel | 259 | cancelFn = cancel | ||
| 198 | mu.Unlock() | 260 | mu.Unlock() | ||
| 199 | 261 | ||||
| 200 | - | turnID := fmt.Sprintf("turn-%d", step) | 262 | + | turnStarted := false |
| 201 | - | em.emit("turn.started", map[string]any{"step": step, "turn_id": turnID}) | |||
| 202 | - | ||||
| 203 | sink := func(ev core.AgentEvent) { | 263 | sink := func(ev core.AgentEvent) { | ||
| 204 | - | em.emit(ev.Type(), subagentEventData(ev)) | 264 | + | data := subagentEventData(ev) |
| 265 | + | if ev.Type() == "user_message" && !turnStarted { | |||
| 266 | + | // Prompt invokes OnMessageAppended before delivering this sink | |||
| 267 | + | // event, so the session row is durable before the lifecycle | |||
| 268 | + | // acknowledgement is written. | |||
| 269 | + | em.emit(ev.Type(), data) | |||
| 270 | + | if !sessionPersistence.failed() { | |||
| 271 | + | em.emit("turn.started", map[string]any{"step": step, "turn_id": turnID, "command_id": commandID, "lifetime_turns": lifetime, "current_run_turns": currentRun}) | |||
| 272 | + | turnStarted = true | |||
| 273 | + | } | |||
| 274 | + | return | |||
| 275 | + | } | |||
| 276 | + | if ev.Type() == "turn_start" { | |||
| 277 | + | // A provider/model-loop turn is nested inside the one delegated | |||
| 278 | + | // message turn admitted above. Mark it on the wire so the | |||
| 279 | + | // supervisor cannot mistake it for a new worker turn. | |||
| 280 | + | data["nested_turn"] = true | |||
| 281 | + | } | |||
| 282 | + | em.emit(ev.Type(), data) | |||
| 205 | } | 283 | } | ||
| 206 | 284 | ||||
| 207 | var persistCompaction func([]provider.Message) error | 285 | var persistCompaction func([]provider.Message) error | ||
| @@ -209,7 +287,12 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) error | |||||
| 209 | persistCompaction = sess.AppendCompaction | 287 | persistCompaction = sess.AppendCompaction | ||
| 210 | } | 288 | } | ||
| 211 | start, err := promptWithContextRecovery(c, ag, prompt, sink, persistCompaction) | 289 | start, err := promptWithContextRecovery(c, ag, prompt, sink, persistCompaction) | ||
| 212 | - | WriteNewTranscript(ag, sess, start) | 290 | + | if sess != nil && ag.OnMessageAppended == nil { |
| 291 | + | WriteNewTranscript(ag, sess, start) | |||
| 292 | + | } | |||
| 293 | + | if sessionErr := sessionPersistence.err(); err == nil && sessionErr != nil { | |||
| 294 | + | err = sessionErr | |||
| 295 | + | } | |||
| 213 | 296 | ||||
| 214 | output := finalAssistantText(ag.Messages()[start:]) | 297 | output := finalAssistantText(ag.Messages()[start:]) | ||
| 215 | resultStatus := "succeeded" | 298 | resultStatus := "succeeded" | ||
| @@ -245,24 +328,26 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) error | |||||
| 245 | // The supervisor treats agent.idle as permission to start a follow-up. | 328 | // The supervisor treats agent.idle as permission to start a follow-up. | ||
| 246 | // Publish it only after the busy gate has opened, otherwise a manager | 329 | // Publish it only after the busy gate has opened, otherwise a manager | ||
| 247 | // can send a turn that the inbox loop drops as still busy. | 330 | // can send a turn that the inbox loop drops as still busy. | ||
| 248 | - | em.emit("agent.idle", map[string]any{"turn_id": turnID}) | 331 | + | em.emit("agent.idle", map[string]any{"turn_id": turnID, "lifetime_turns": lifetime, "current_run_turns": currentRun}) |
| 249 | } | 332 | } | ||
| 250 | 333 | ||||
| 251 | - | launchTurn := func(prompt string) { | 334 | + | launchTurn := func(prompt string, newRun bool, commandID string) bool { |
| 252 | mu.Lock() | 335 | mu.Lock() | ||
| 253 | - | if closing { | 336 | + | if closing || busyTurn || turnPending { |
| 254 | mu.Unlock() | 337 | mu.Unlock() | ||
| 255 | - | return | 338 | + | em.emit("error", map[string]any{"code": "turn_rejected", "reason": "busy", "command_id": commandID, "message": "worker is busy"}) |
| 339 | + | return false | |||
| 256 | } | 340 | } | ||
| 257 | - | // Register the goroutine before launching it. Shutdown can then | 341 | + | // Reserve the worker before launching the goroutine. A second command |
| 258 | - | // close the gate and wait for every turn that was admitted, even | 342 | + | // cannot be accepted in the gap before runOne acquires mu. |
| 259 | - | // if runOne has not acquired mu and set busyTurn yet. | 343 | + | turnPending = true |
| 260 | turnWG.Add(1) | 344 | turnWG.Add(1) | ||
| 261 | mu.Unlock() | 345 | mu.Unlock() | ||
| 262 | go func() { | 346 | go func() { | ||
| 263 | defer turnWG.Done() | 347 | defer turnWG.Done() | ||
| 264 | - | runOne(prompt) | 348 | + | runOne(prompt, newRun, commandID) |
| 265 | }() | 349 | }() | ||
| 350 | + | return true | |||
| 266 | } | 351 | } | ||
| 267 | 352 | ||||
| 268 | waitForActiveTurn := func(cancel bool) { | 353 | waitForActiveTurn := func(cancel bool) { | ||
| @@ -287,7 +372,7 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) error | |||||
| 287 | // "starts working" the moment it boots, matching what users | 372 | // "starts working" the moment it boots, matching what users | ||
| 288 | // expect from `/subagents new <task>`. | 373 | // expect from `/subagents new <task>`. | ||
| 289 | if args.Prompt != "" { | 374 | if args.Prompt != "" { | ||
| 290 | - | launchTurn(args.Prompt) | 375 | + | launchTurn(args.Prompt, false, "") |
| 291 | } | 376 | } | ||
| 292 | 377 | ||||
| 293 | // Inbox loop: one supervisor message at a time. We don't spawn | 378 | // Inbox loop: one supervisor message at a time. We don't spawn | ||
| @@ -333,10 +418,20 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) error | |||||
| 333 | case subagents.CommandTurnStart: | 418 | case subagents.CommandTurnStart: | ||
| 334 | var payload subagents.TurnStartPayload | 419 | var payload subagents.TurnStartPayload | ||
| 335 | if err := command.DecodePayload(&payload); err != nil { | 420 | if err := command.DecodePayload(&payload); err != nil { | ||
| 336 | - | em.emit("error", map[string]any{"message": "invalid turn.start payload"}) | 421 | + | em.emit("error", map[string]any{"code": "turn_rejected", "reason": "invalid_payload", "command_id": command.MessageID, "message": "invalid turn.start payload"}) |
| 337 | continue | 422 | continue | ||
| 338 | } | 423 | } | ||
| 339 | - | launchTurn(payload.Prompt) | 424 | + | mu.Lock() |
| 425 | + | _, duplicate := seenCommands[command.MessageID] | |||
| 426 | + | mu.Unlock() | |||
| 427 | + | if duplicate && command.MessageID != "" { | |||
| 428 | + | continue | |||
| 429 | + | } | |||
| 430 | + | if launchTurn(payload.Prompt, payload.NewRun, command.MessageID) && command.MessageID != "" { | |||
| 431 | + | mu.Lock() | |||
| 432 | + | seenCommands[command.MessageID] = struct{}{} | |||
| 433 | + | mu.Unlock() | |||
| 434 | + | } | |||
| 340 | default: | 435 | default: | ||
| 341 | // Unknown commands are rejected rather than interpreted as | 436 | // Unknown commands are rejected rather than interpreted as | ||
| 342 | // child-originated control requests. | 437 | // child-originated control requests. | ||
| @@ -555,6 +650,9 @@ func initialTurnNumber(path string) int { | |||||
| 555 | } | 650 | } | ||
| 556 | maxStep := 0 | 651 | maxStep := 0 | ||
| 557 | for _, ev := range events { | 652 | for _, ev := range events { | ||
| 653 | + | if !subagents.IsDelegatedTurnStart(ev) { | |||
| 654 | + | continue | |||
| 655 | + | } | |||
| 558 | if step, ok := ev.Data["step"].(float64); ok && int(step) > maxStep { | 656 | if step, ok := ev.Data["step"].(float64); ok && int(step) > maxStep { | ||
| 559 | maxStep = int(step) | 657 | maxStep = int(step) | ||
| 560 | } | 658 | } | ||
M packages/agent/subagent_worker_test.go +46 -0
| @@ -526,3 +526,49 @@ func TestSupervisorEmitterStdoutShapeMatchesSupervisorParser(t *testing.T) { | |||
| 526 | 526 | t.Errorf("payload step field missing or wrong: %v", object["payload"]) | |
| 527 | 527 | } | |
| 528 | 528 | } | |
| 529 | + | ||
| 530 | + | func TestInitialTurnNumberIgnoresNestedProviderTurns(t *testing.T) { | |
| 531 | + | path := filepath.Join(t.TempDir(), "events.jsonl") | |
| 532 | + | log, err := subagents.OpenEventLog(path) | |
| 533 | + | if err != nil { | |
| 534 | + | t.Fatal(err) | |
| 535 | + | } | |
| 536 | + | if err := log.Append(subagents.NewEvent(subagents.EventTurnStarted, map[string]any{ | |
| 537 | + | "step": float64(1), "lifetime_turns": 1, "current_run_turns": 1, | |
| 538 | + | })); err != nil { | |
| 539 | + | t.Fatal(err) | |
| 540 | + | } | |
| 541 | + | if err := log.Append(subagents.NewEvent(subagents.EventTurnStarted, map[string]any{ | |
| 542 | + | "step": float64(99), "nested_turn": true, | |
| 543 | + | })); err != nil { | |
| 544 | + | t.Fatal(err) | |
| 545 | + | } | |
| 546 | + | if err := log.Close(); err != nil { | |
| 547 | + | t.Fatal(err) | |
| 548 | + | } | |
| 549 | + | if got := initialTurnNumber(path); got != 1 { | |
| 550 | + | t.Fatalf("initial turn number = %d, want 1", got) | |
| 551 | + | } | |
| 552 | + | } | |
| 553 | + | ||
| 554 | + | func TestWorkerTurnBudgetFreshAllowanceOnlyForExplicitRun(t *testing.T) { | |
| 555 | + | var budget workerTurnBudget | |
| 556 | + | ||
| 557 | + | if _, lifetime, current, admitted := budget.start(2, false); !admitted || lifetime != 1 || current != 1 { | |
| 558 | + | t.Fatalf("first turn = (%d, %d, %v), want (1, 1, true)", lifetime, current, admitted) | |
| 559 | + | } | |
| 560 | + | // Provider retries, model loops, and compaction stay inside the admitted | |
| 561 | + | // turn and therefore do not call start or consume another allowance. | |
| 562 | + | if budget.lifetime != 1 || budget.current != 1 { | |
| 563 | + | t.Fatalf("nested work changed budget to (lifetime=%d, current=%d)", budget.lifetime, budget.current) | |
| 564 | + | } | |
| 565 | + | if _, lifetime, current, admitted := budget.start(2, false); !admitted || lifetime != 2 || current != 2 { | |
| 566 | + | t.Fatalf("second turn = (%d, %d, %v), want (2, 2, true)", lifetime, current, admitted) | |
| 567 | + | } | |
| 568 | + | if _, lifetime, current, admitted := budget.start(2, false); admitted || lifetime != 2 || current != 2 { | |
| 569 | + | t.Fatalf("exhausted turn = (%d, %d, %v), want (2, 2, false)", lifetime, current, admitted) | |
| 570 | + | } | |
| 571 | + | if _, lifetime, current, admitted := budget.start(2, true); !admitted || lifetime != 3 || current != 1 { | |
| 572 | + | t.Fatalf("explicit resumed turn = (%d, %d, %v), want (3, 1, true)", lifetime, current, admitted) | |
| 573 | + | } | |
| 574 | + | } | |
| @@ -526,3 +526,49 @@ func TestSupervisorEmitterStdoutShapeMatchesSupervisorParser(t *testing.T) { | |||||
| 526 | t.Errorf("payload step field missing or wrong: %v", object["payload"]) | 526 | t.Errorf("payload step field missing or wrong: %v", object["payload"]) | ||
| 527 | } | 527 | } | ||
| 528 | } | 528 | } | ||
| 529 | + | ||||
| 530 | + | func TestInitialTurnNumberIgnoresNestedProviderTurns(t *testing.T) { | |||
| 531 | + | path := filepath.Join(t.TempDir(), "events.jsonl") | |||
| 532 | + | log, err := subagents.OpenEventLog(path) | |||
| 533 | + | if err != nil { | |||
| 534 | + | t.Fatal(err) | |||
| 535 | + | } | |||
| 536 | + | if err := log.Append(subagents.NewEvent(subagents.EventTurnStarted, map[string]any{ | |||
| 537 | + | "step": float64(1), "lifetime_turns": 1, "current_run_turns": 1, | |||
| 538 | + | })); err != nil { | |||
| 539 | + | t.Fatal(err) | |||
| 540 | + | } | |||
| 541 | + | if err := log.Append(subagents.NewEvent(subagents.EventTurnStarted, map[string]any{ | |||
| 542 | + | "step": float64(99), "nested_turn": true, | |||
| 543 | + | })); err != nil { | |||
| 544 | + | t.Fatal(err) | |||
| 545 | + | } | |||
| 546 | + | if err := log.Close(); err != nil { | |||
| 547 | + | t.Fatal(err) | |||
| 548 | + | } | |||
| 549 | + | if got := initialTurnNumber(path); got != 1 { | |||
| 550 | + | t.Fatalf("initial turn number = %d, want 1", got) | |||
| 551 | + | } | |||
| 552 | + | } | |||
| 553 | + | ||||
| 554 | + | func TestWorkerTurnBudgetFreshAllowanceOnlyForExplicitRun(t *testing.T) { | |||
| 555 | + | var budget workerTurnBudget | |||
| 556 | + | ||||
| 557 | + | if _, lifetime, current, admitted := budget.start(2, false); !admitted || lifetime != 1 || current != 1 { | |||
| 558 | + | t.Fatalf("first turn = (%d, %d, %v), want (1, 1, true)", lifetime, current, admitted) | |||
| 559 | + | } | |||
| 560 | + | // Provider retries, model loops, and compaction stay inside the admitted | |||
| 561 | + | // turn and therefore do not call start or consume another allowance. | |||
| 562 | + | if budget.lifetime != 1 || budget.current != 1 { | |||
| 563 | + | t.Fatalf("nested work changed budget to (lifetime=%d, current=%d)", budget.lifetime, budget.current) | |||
| 564 | + | } | |||
| 565 | + | if _, lifetime, current, admitted := budget.start(2, false); !admitted || lifetime != 2 || current != 2 { | |||
| 566 | + | t.Fatalf("second turn = (%d, %d, %v), want (2, 2, true)", lifetime, current, admitted) | |||
| 567 | + | } | |||
| 568 | + | if _, lifetime, current, admitted := budget.start(2, false); admitted || lifetime != 2 || current != 2 { | |||
| 569 | + | t.Fatalf("exhausted turn = (%d, %d, %v), want (2, 2, false)", lifetime, current, admitted) | |||
| 570 | + | } | |||
| 571 | + | if _, lifetime, current, admitted := budget.start(2, true); !admitted || lifetime != 3 || current != 1 { | |||
| 572 | + | t.Fatalf("explicit resumed turn = (%d, %d, %v), want (3, 1, true)", lifetime, current, admitted) | |||
| 573 | + | } | |||
| 574 | + | } | |||
A packages/agent/subagents/ack_test.go +40 -0
| @@ -0,0 +1,40 @@ | |||
| 1 | + | package subagents | |
| 2 | + | ||
| 3 | + | import ( | |
| 4 | + | "testing" | |
| 5 | + | "time" | |
| 6 | + | ) | |
| 7 | + | ||
| 8 | + | func TestResumePromptAcknowledgementRequiresDurableUserMessageAndTurnStart(t *testing.T) { | |
| 9 | + | acceptedAt := time.Now().Add(-time.Second) | |
| 10 | + | prompt := "follow-up prompt" | |
| 11 | + | user := NewEvent("user_message", map[string]any{ | |
| 12 | + | "content": []any{map[string]any{"type": "text", "text": prompt}}, | |
| 13 | + | }) | |
| 14 | + | started := NewEvent(EventTurnStarted, map[string]any{ | |
| 15 | + | "lifetime_turns": 2, | |
| 16 | + | "current_run_turns": 1, | |
| 17 | + | }) | |
| 18 | + | ||
| 19 | + | if resumePromptAcknowledged([]Event{started}, prompt, acceptedAt) { | |
| 20 | + | t.Fatal("turn.started alone acknowledged a pending prompt") | |
| 21 | + | } | |
| 22 | + | if resumePromptAcknowledged([]Event{user}, prompt, acceptedAt) { | |
| 23 | + | t.Fatal("crash point after durable user message acknowledged before turn.started") | |
| 24 | + | } | |
| 25 | + | if !resumePromptAcknowledged([]Event{user, started}, prompt, acceptedAt) { | |
| 26 | + | t.Fatal("matching durable user message followed by delegated turn.started was not acknowledged") | |
| 27 | + | } | |
| 28 | + | } | |
| 29 | + | ||
| 30 | + | func TestResumePromptAcknowledgementIgnoresNestedTurnStart(t *testing.T) { | |
| 31 | + | acceptedAt := time.Now().Add(-time.Second) | |
| 32 | + | prompt := "follow-up prompt" | |
| 33 | + | user := NewEvent("user_message", map[string]any{ | |
| 34 | + | "content": []any{map[string]any{"type": "text", "text": prompt}}, | |
| 35 | + | }) | |
| 36 | + | nested := NewEvent(EventTurnStarted, map[string]any{"nested_turn": true}) | |
| 37 | + | if resumePromptAcknowledged([]Event{user, nested}, prompt, acceptedAt) { | |
| 38 | + | t.Fatal("nested turn.started acknowledged a pending delegated prompt") | |
| 39 | + | } | |
| 40 | + | } | |
| @@ -0,0 +1,40 @@ | |||||
| 1 | + | package subagents | |||
| 2 | + | ||||
| 3 | + | import ( | |||
| 4 | + | "testing" | |||
| 5 | + | "time" | |||
| 6 | + | ) | |||
| 7 | + | ||||
| 8 | + | func TestResumePromptAcknowledgementRequiresDurableUserMessageAndTurnStart(t *testing.T) { | |||
| 9 | + | acceptedAt := time.Now().Add(-time.Second) | |||
| 10 | + | prompt := "follow-up prompt" | |||
| 11 | + | user := NewEvent("user_message", map[string]any{ | |||
| 12 | + | "content": []any{map[string]any{"type": "text", "text": prompt}}, | |||
| 13 | + | }) | |||
| 14 | + | started := NewEvent(EventTurnStarted, map[string]any{ | |||
| 15 | + | "lifetime_turns": 2, | |||
| 16 | + | "current_run_turns": 1, | |||
| 17 | + | }) | |||
| 18 | + | ||||
| 19 | + | if resumePromptAcknowledged([]Event{started}, prompt, acceptedAt) { | |||
| 20 | + | t.Fatal("turn.started alone acknowledged a pending prompt") | |||
| 21 | + | } | |||
| 22 | + | if resumePromptAcknowledged([]Event{user}, prompt, acceptedAt) { | |||
| 23 | + | t.Fatal("crash point after durable user message acknowledged before turn.started") | |||
| 24 | + | } | |||
| 25 | + | if !resumePromptAcknowledged([]Event{user, started}, prompt, acceptedAt) { | |||
| 26 | + | t.Fatal("matching durable user message followed by delegated turn.started was not acknowledged") | |||
| 27 | + | } | |||
| 28 | + | } | |||
| 29 | + | ||||
| 30 | + | func TestResumePromptAcknowledgementIgnoresNestedTurnStart(t *testing.T) { | |||
| 31 | + | acceptedAt := time.Now().Add(-time.Second) | |||
| 32 | + | prompt := "follow-up prompt" | |||
| 33 | + | user := NewEvent("user_message", map[string]any{ | |||
| 34 | + | "content": []any{map[string]any{"type": "text", "text": prompt}}, | |||
| 35 | + | }) | |||
| 36 | + | nested := NewEvent(EventTurnStarted, map[string]any{"nested_turn": true}) | |||
| 37 | + | if resumePromptAcknowledged([]Event{user, nested}, prompt, acceptedAt) { | |||
| 38 | + | t.Fatal("nested turn.started acknowledged a pending delegated prompt") | |||
| 39 | + | } | |||
| 40 | + | } | |||
M packages/agent/subagents/agent.go +204 -7
| @@ -7,6 +7,8 @@ import ( | |||
| 7 | 7 | "strings" | |
| 8 | 8 | "sync" | |
| 9 | 9 | "time" | |
| 10 | + | ||
| 11 | + | "github.com/google/uuid" | |
| 10 | 12 | ) | |
| 11 | 13 | ||
| 12 | 14 | // Agent is one supervised task. Public fields are immutable after | |
| @@ -17,6 +19,12 @@ type turnEndNotice struct { | |||
| 17 | 19 | errMsg string | |
| 18 | 20 | } | |
| 19 | 21 | ||
| 22 | + | type queuedResumePrompt struct { | |
| 23 | + | Prompt string `json:"prompt"` | |
| 24 | + | AcceptedAt time.Time `json:"accepted_at"` | |
| 25 | + | CommandID string `json:"command_id,omitempty"` | |
| 26 | + | } | |
| 27 | + | ||
| 20 | 28 | type Agent struct { | |
| 21 | 29 | ID string | |
| 22 | 30 | Task string | |
| @@ -36,6 +44,14 @@ type Agent struct { | |||
| 36 | 44 | WorkspaceBase string | |
| 37 | 45 | WorkspaceCapture CaptureMode | |
| 38 | 46 | MaxTurns int | |
| 47 | + | ||
| 48 | + | // LifetimeTurns counts every accepted message turn for this worker across | |
| 49 | + | // all explicit runs. CurrentRunTurns resets only when an explicit | |
| 50 | + | // subagent_resume follow-up starts a new run; retries, model loops, | |
| 51 | + | // compaction, and cancellation do not reset it. | |
| 52 | + | LifetimeTurns int | |
| 53 | + | CurrentRunTurns int | |
| 54 | + | ||
| 39 | 55 | // Timeout is the effective per-agent lifetime, persisted so a resumed | |
| 40 | 56 | // worker keeps the timeout selected at its spawn boundary. | |
| 41 | 57 | Timeout time.Duration | |
| @@ -80,12 +96,15 @@ type Agent struct { | |||
| 80 | 96 | // Not persisted — every Resume sets it explicitly. | |
| 81 | 97 | Resuming bool | |
| 82 | 98 | ||
| 83 | - | // resumePromptText is an optional manager follow-up passed to a resumed | |
| 84 | - | // child as its initial prompt. It remains in durable metadata until the | |
| 85 | - | // child reports turn.started, so a host exit while the worker is queued | |
| 86 | - | // cannot lose it. Access it through ResumePromptInfo. | |
| 87 | - | resumePromptText string | |
| 88 | - | resumePromptAt time.Time | |
| 99 | + | // resumePromptText is the manager follow-up currently handed to the | |
| 100 | + | // child. It remains in durable metadata until the child reports | |
| 101 | + | // turn.started, so a host exit while the worker is queued cannot lose it. | |
| 102 | + | // Additional follow-ups accepted while the child is busy stay in | |
| 103 | + | // resumePromptQueue until the next agent idle turn. | |
| 104 | + | resumePromptText string | |
| 105 | + | resumePromptAt time.Time | |
| 106 | + | resumePromptID string | |
| 107 | + | resumePromptQueue []queuedResumePrompt | |
| 89 | 108 | ||
| 90 | 109 | // InboxPath is the unix socket the child agent listens on for | |
| 91 | 110 | // follow-up prompts and control messages. The supervisor opens | |
| @@ -142,10 +161,12 @@ type Agent struct { | |||
| 142 | 161 | outputTruncated bool | |
| 143 | 162 | maxOutputBytes int | |
| 144 | 163 | maxOutputLines int | |
| 145 | - | persistFn func(*Agent) | |
| 164 | + | persistFn func(*Agent) error | |
| 165 | + | persistMu sync.Mutex | |
| 146 | 166 | stateDir string | |
| 147 | 167 | workspaceCleanup func() error | |
| 148 | 168 | workspaceCapture func() (WorkspaceCapture, error) | |
| 169 | + | onTurnIdle func() | |
| 149 | 170 | ||
| 150 | 171 | // OnTurnEnd, if set, fires once per prompt-level turn_end event | |
| 151 | 172 | // emitted by the subagent daemon wrapper. Provider/tool-loop | |
| @@ -230,6 +251,66 @@ func (a *Agent) AttemptValue() int { | |||
| 230 | 251 | return a.Attempt | |
| 231 | 252 | } | |
| 232 | 253 | ||
| 254 | + | // LifetimeTurnsValue returns the number of accepted message turns across all | |
| 255 | + | // explicit runs of this worker. | |
| 256 | + | func (a *Agent) LifetimeTurnsValue() int { | |
| 257 | + | a.lifecycleMu.Lock() | |
| 258 | + | defer a.lifecycleMu.Unlock() | |
| 259 | + | return a.LifetimeTurns | |
| 260 | + | } | |
| 261 | + | ||
| 262 | + | // CurrentRunTurnsValue returns the number of accepted message turns in the | |
| 263 | + | // current explicit run. | |
| 264 | + | func (a *Agent) CurrentRunTurnsValue() int { | |
| 265 | + | a.lifecycleMu.Lock() | |
| 266 | + | defer a.lifecycleMu.Unlock() | |
| 267 | + | return a.CurrentRunTurns | |
| 268 | + | } | |
| 269 | + | ||
| 270 | + | func (a *Agent) setTurnCounts(lifetime, currentRun int) { | |
| 271 | + | if lifetime < 0 { | |
| 272 | + | lifetime = 0 | |
| 273 | + | } | |
| 274 | + | if currentRun < 0 { | |
| 275 | + | currentRun = 0 | |
| 276 | + | } | |
| 277 | + | a.lifecycleMu.Lock() | |
| 278 | + | a.LifetimeTurns = lifetime | |
| 279 | + | a.CurrentRunTurns = currentRun | |
| 280 | + | a.updatedAt = time.Now() | |
| 281 | + | a.lifecycleMu.Unlock() | |
| 282 | + | } | |
| 283 | + | ||
| 284 | + | func (a *Agent) incrementTurnCounts() (lifetime, currentRun int) { | |
| 285 | + | a.lifecycleMu.Lock() | |
| 286 | + | a.LifetimeTurns++ | |
| 287 | + | a.CurrentRunTurns++ | |
| 288 | + | a.updatedAt = time.Now() | |
| 289 | + | lifetime = a.LifetimeTurns | |
| 290 | + | currentRun = a.CurrentRunTurns | |
| 291 | + | a.lifecycleMu.Unlock() | |
| 292 | + | return lifetime, currentRun | |
| 293 | + | } | |
| 294 | + | ||
| 295 | + | func (a *Agent) resetCurrentRunTurns() (previous int) { | |
| 296 | + | a.lifecycleMu.Lock() | |
| 297 | + | previous = a.CurrentRunTurns | |
| 298 | + | a.CurrentRunTurns = 0 | |
| 299 | + | a.updatedAt = time.Now() | |
| 300 | + | a.lifecycleMu.Unlock() | |
| 301 | + | return previous | |
| 302 | + | } | |
| 303 | + | ||
| 304 | + | func (a *Agent) setCurrentRunTurns(currentRun int) { | |
| 305 | + | if currentRun < 0 { | |
| 306 | + | currentRun = 0 | |
| 307 | + | } | |
| 308 | + | a.lifecycleMu.Lock() | |
| 309 | + | a.CurrentRunTurns = currentRun | |
| 310 | + | a.updatedAt = time.Now() | |
| 311 | + | a.lifecycleMu.Unlock() | |
| 312 | + | } | |
| 313 | + | ||
| 233 | 314 | // ProcessPIDValue returns the current worker pid, when known. | |
| 234 | 315 | func (a *Agent) ProcessPIDValue() int { | |
| 235 | 316 | a.lifecycleMu.Lock() | |
| @@ -406,9 +487,16 @@ func (a *Agent) setResumePrompt(prompt string, acceptedAt time.Time) (string, ti | |||
| 406 | 487 | previousPrompt, previousAcceptedAt := a.resumePromptText, a.resumePromptAt | |
| 407 | 488 | a.resumePromptText = prompt | |
| 408 | 489 | a.resumePromptAt = acceptedAt | |
| 490 | + | a.resumePromptID = uuid.NewString() | |
| 409 | 491 | return previousPrompt, previousAcceptedAt | |
| 410 | 492 | } | |
| 411 | 493 | ||
| 494 | + | func (a *Agent) resumePromptCommandID() string { | |
| 495 | + | a.mu.Lock() | |
| 496 | + | defer a.mu.Unlock() | |
| 497 | + | return a.resumePromptID | |
| 498 | + | } | |
| 499 | + | ||
| 412 | 500 | func (a *Agent) clearResumePrompt() bool { | |
| 413 | 501 | a.mu.Lock() | |
| 414 | 502 | defer a.mu.Unlock() | |
| @@ -417,9 +505,106 @@ func (a *Agent) clearResumePrompt() bool { | |||
| 417 | 505 | } | |
| 418 | 506 | a.resumePromptText = "" | |
| 419 | 507 | a.resumePromptAt = time.Time{} | |
| 508 | + | a.resumePromptID = "" | |
| 420 | 509 | return true | |
| 421 | 510 | } | |
| 422 | 511 | ||
| 512 | + | func (a *Agent) queueResumePrompt(prompt string, acceptedAt time.Time) { | |
| 513 | + | a.mu.Lock() | |
| 514 | + | a.resumePromptQueue = append(a.resumePromptQueue, queuedResumePrompt{Prompt: prompt, AcceptedAt: acceptedAt, CommandID: uuid.NewString()}) | |
| 515 | + | a.mu.Unlock() | |
| 516 | + | } | |
| 517 | + | ||
| 518 | + | func (a *Agent) removeQueuedResumePrompt(prompt string, acceptedAt time.Time) bool { | |
| 519 | + | a.mu.Lock() | |
| 520 | + | defer a.mu.Unlock() | |
| 521 | + | for i := len(a.resumePromptQueue) - 1; i >= 0; i-- { | |
| 522 | + | queued := a.resumePromptQueue[i] | |
| 523 | + | if queued.Prompt == prompt && queued.AcceptedAt.Equal(acceptedAt) { | |
| 524 | + | a.resumePromptQueue = append(a.resumePromptQueue[:i], a.resumePromptQueue[i+1:]...) | |
| 525 | + | return true | |
| 526 | + | } | |
| 527 | + | } | |
| 528 | + | return false | |
| 529 | + | } | |
| 530 | + | ||
| 531 | + | func (a *Agent) resumePromptQueueSnapshot() []queuedResumePrompt { | |
| 532 | + | a.mu.Lock() | |
| 533 | + | defer a.mu.Unlock() | |
| 534 | + | return append([]queuedResumePrompt(nil), a.resumePromptQueue...) | |
| 535 | + | } | |
| 536 | + | ||
| 537 | + | func (a *Agent) hasQueuedResumePrompt() bool { | |
| 538 | + | a.mu.Lock() | |
| 539 | + | defer a.mu.Unlock() | |
| 540 | + | return len(a.resumePromptQueue) != 0 | |
| 541 | + | } | |
| 542 | + | ||
| 543 | + | // startQueuedResumePrompt moves the oldest active follow-up into the | |
| 544 | + | // in-flight prompt slot. The caller must persist the returned state before | |
| 545 | + | // delivering the command so a host exit cannot lose the prompt. | |
| 546 | + | func (a *Agent) startQueuedResumePrompt() (string, time.Time, string, bool) { | |
| 547 | + | a.mu.Lock() | |
| 548 | + | defer a.mu.Unlock() | |
| 549 | + | if a.resumePromptText != "" || len(a.resumePromptQueue) == 0 { | |
| 550 | + | return "", time.Time{}, "", false | |
| 551 | + | } | |
| 552 | + | queued := a.resumePromptQueue[0] | |
| 553 | + | a.resumePromptQueue = a.resumePromptQueue[1:] | |
| 554 | + | a.resumePromptText = queued.Prompt | |
| 555 | + | a.resumePromptAt = queued.AcceptedAt | |
| 556 | + | a.resumePromptID = queued.CommandID | |
| 557 | + | if a.resumePromptID == "" { | |
| 558 | + | a.resumePromptID = uuid.NewString() | |
| 559 | + | } | |
| 560 | + | return queued.Prompt, queued.AcceptedAt, a.resumePromptID, true | |
| 561 | + | } | |
| 562 | + | ||
| 563 | + | func (a *Agent) restoreResumePromptToQueue(prompt string, acceptedAt time.Time) bool { | |
| 564 | + | a.mu.Lock() | |
| 565 | + | defer a.mu.Unlock() | |
| 566 | + | if a.resumePromptText != prompt || !a.resumePromptAt.Equal(acceptedAt) { | |
| 567 | + | return false | |
| 568 | + | } | |
| 569 | + | commandID := a.resumePromptID | |
| 570 | + | if commandID == "" { | |
| 571 | + | commandID = uuid.NewString() | |
| 572 | + | } | |
| 573 | + | a.resumePromptText = "" | |
| 574 | + | a.resumePromptAt = time.Time{} | |
| 575 | + | a.resumePromptID = "" | |
| 576 | + | a.resumePromptQueue = append([]queuedResumePrompt{{Prompt: prompt, AcceptedAt: acceptedAt, CommandID: commandID}}, a.resumePromptQueue...) | |
| 577 | + | return true | |
| 578 | + | } | |
| 579 | + | ||
| 580 | + | func (a *Agent) rejectActiveResumePrompt() bool { | |
| 581 | + | a.mu.Lock() | |
| 582 | + | defer a.mu.Unlock() | |
| 583 | + | if a.resumePromptText == "" { | |
| 584 | + | return false | |
| 585 | + | } | |
| 586 | + | a.resumePromptQueue = append([]queuedResumePrompt{{Prompt: a.resumePromptText, AcceptedAt: a.resumePromptAt, CommandID: a.resumePromptID}}, a.resumePromptQueue...) | |
| 587 | + | a.resumePromptText = "" | |
| 588 | + | a.resumePromptAt = time.Time{} | |
| 589 | + | a.resumePromptID = "" | |
| 590 | + | return true | |
| 591 | + | } | |
| 592 | + | ||
| 593 | + | func (a *Agent) setOnTurnIdle(fn func()) { | |
| 594 | + | a.mu.Lock() | |
| 595 | + | a.onTurnIdle = fn | |
| 596 | + | a.mu.Unlock() | |
| 597 | + | } | |
| 598 | + | ||
| 599 | + | func (a *Agent) notifyTurnIdle() { | |
| 600 | + | a.mu.Lock() | |
| 601 | + | fn := a.onTurnIdle | |
| 602 | + | a.mu.Unlock() | |
| 603 | + | if fn != nil { | |
| 604 | + | fn() | |
| 605 | + | } | |
| 606 | + | } | |
| 607 | + | ||
| 423 | 608 | func (a *Agent) SetOnTurnEnd(fn func(step int, errMsg string)) { | |
| 424 | 609 | a.mu.Lock() | |
| 425 | 610 | a.OnTurnEnd = fn | |
| @@ -433,6 +618,18 @@ func (a *Agent) SetOnTurnEnd(fn func(step int, errMsg string)) { | |||
| 433 | 618 | } | |
| 434 | 619 | } | |
| 435 | 620 | ||
| 621 | + | func (a *Agent) recordPersistenceError(err error) { | |
| 622 | + | if a == nil || err == nil { | |
| 623 | + | return | |
| 624 | + | } | |
| 625 | + | a.mu.Lock() | |
| 626 | + | a.lastErr = fmt.Errorf("subagent metadata persistence: %w", err) | |
| 627 | + | if a.status != StatusDone && a.status != StatusFailed && a.status != StatusKilled { | |
| 628 | + | a.activity = "metadata error: " + truncate(err.Error(), 120) | |
| 629 | + | } | |
| 630 | + | a.mu.Unlock() | |
| 631 | + | } | |
| 632 | + | ||
| 436 | 633 | func (a *Agent) setActivity(msg string) { | |
| 437 | 634 | a.mu.Lock() | |
| 438 | 635 | a.activity = strings.TrimSpace(msg) | |
| @@ -7,6 +7,8 @@ import ( | |||||
| 7 | "strings" | 7 | "strings" | ||
| 8 | "sync" | 8 | "sync" | ||
| 9 | "time" | 9 | "time" | ||
| 10 | + | ||||
| 11 | + | "github.com/google/uuid" | |||
| 10 | ) | 12 | ) | ||
| 11 | 13 | ||||
| 12 | // Agent is one supervised task. Public fields are immutable after | 14 | // Agent is one supervised task. Public fields are immutable after | ||
| @@ -17,6 +19,12 @@ type turnEndNotice struct { | |||||
| 17 | errMsg string | 19 | errMsg string | ||
| 18 | } | 20 | } | ||
| 19 | 21 | ||||
| 22 | + | type queuedResumePrompt struct { | |||
| 23 | + | Prompt string `json:"prompt"` | |||
| 24 | + | AcceptedAt time.Time `json:"accepted_at"` | |||
| 25 | + | CommandID string `json:"command_id,omitempty"` | |||
| 26 | + | } | |||
| 27 | + | ||||
| 20 | type Agent struct { | 28 | type Agent struct { | ||
| 21 | ID string | 29 | ID string | ||
| 22 | Task string | 30 | Task string | ||
| @@ -36,6 +44,14 @@ type Agent struct { | |||||
| 36 | WorkspaceBase string | 44 | WorkspaceBase string | ||
| 37 | WorkspaceCapture CaptureMode | 45 | WorkspaceCapture CaptureMode | ||
| 38 | MaxTurns int | 46 | MaxTurns int | ||
| 47 | + | ||||
| 48 | + | // LifetimeTurns counts every accepted message turn for this worker across | |||
| 49 | + | // all explicit runs. CurrentRunTurns resets only when an explicit | |||
| 50 | + | // subagent_resume follow-up starts a new run; retries, model loops, | |||
| 51 | + | // compaction, and cancellation do not reset it. | |||
| 52 | + | LifetimeTurns int | |||
| 53 | + | CurrentRunTurns int | |||
| 54 | + | ||||
| 39 | // Timeout is the effective per-agent lifetime, persisted so a resumed | 55 | // Timeout is the effective per-agent lifetime, persisted so a resumed | ||
| 40 | // worker keeps the timeout selected at its spawn boundary. | 56 | // worker keeps the timeout selected at its spawn boundary. | ||
| 41 | Timeout time.Duration | 57 | Timeout time.Duration | ||
| @@ -80,12 +96,15 @@ type Agent struct { | |||||
| 80 | // Not persisted — every Resume sets it explicitly. | 96 | // Not persisted — every Resume sets it explicitly. | ||
| 81 | Resuming bool | 97 | Resuming bool | ||
| 82 | 98 | ||||
| 83 | - | // resumePromptText is an optional manager follow-up passed to a resumed | 99 | + | // resumePromptText is the manager follow-up currently handed to the |
| 84 | - | // child as its initial prompt. It remains in durable metadata until the | 100 | + | // child. It remains in durable metadata until the child reports |
| 85 | - | // child reports turn.started, so a host exit while the worker is queued | 101 | + | // turn.started, so a host exit while the worker is queued cannot lose it. |
| 86 | - | // cannot lose it. Access it through ResumePromptInfo. | 102 | + | // Additional follow-ups accepted while the child is busy stay in |
| 87 | - | resumePromptText string | 103 | + | // resumePromptQueue until the next agent idle turn. |
| 88 | - | resumePromptAt time.Time | 104 | + | resumePromptText string |
| 105 | + | resumePromptAt time.Time | |||
| 106 | + | resumePromptID string | |||
| 107 | + | resumePromptQueue []queuedResumePrompt | |||
| 89 | 108 | ||||
| 90 | // InboxPath is the unix socket the child agent listens on for | 109 | // InboxPath is the unix socket the child agent listens on for | ||
| 91 | // follow-up prompts and control messages. The supervisor opens | 110 | // follow-up prompts and control messages. The supervisor opens | ||
| @@ -142,10 +161,12 @@ type Agent struct { | |||||
| 142 | outputTruncated bool | 161 | outputTruncated bool | ||
| 143 | maxOutputBytes int | 162 | maxOutputBytes int | ||
| 144 | maxOutputLines int | 163 | maxOutputLines int | ||
| 145 | - | persistFn func(*Agent) | 164 | + | persistFn func(*Agent) error |
| 165 | + | persistMu sync.Mutex | |||
| 146 | stateDir string | 166 | stateDir string | ||
| 147 | workspaceCleanup func() error | 167 | workspaceCleanup func() error | ||
| 148 | workspaceCapture func() (WorkspaceCapture, error) | 168 | workspaceCapture func() (WorkspaceCapture, error) | ||
| 169 | + | onTurnIdle func() | |||
| 149 | 170 | ||||
| 150 | // OnTurnEnd, if set, fires once per prompt-level turn_end event | 171 | // OnTurnEnd, if set, fires once per prompt-level turn_end event | ||
| 151 | // emitted by the subagent daemon wrapper. Provider/tool-loop | 172 | // emitted by the subagent daemon wrapper. Provider/tool-loop | ||
| @@ -230,6 +251,66 @@ func (a *Agent) AttemptValue() int { | |||||
| 230 | return a.Attempt | 251 | return a.Attempt | ||
| 231 | } | 252 | } | ||
| 232 | 253 | ||||
| 254 | + | // LifetimeTurnsValue returns the number of accepted message turns across all | |||
| 255 | + | // explicit runs of this worker. | |||
| 256 | + | func (a *Agent) LifetimeTurnsValue() int { | |||
| 257 | + | a.lifecycleMu.Lock() | |||
| 258 | + | defer a.lifecycleMu.Unlock() | |||
| 259 | + | return a.LifetimeTurns | |||
| 260 | + | } | |||
| 261 | + | ||||
| 262 | + | // CurrentRunTurnsValue returns the number of accepted message turns in the | |||
| 263 | + | // current explicit run. | |||
| 264 | + | func (a *Agent) CurrentRunTurnsValue() int { | |||
| 265 | + | a.lifecycleMu.Lock() | |||
| 266 | + | defer a.lifecycleMu.Unlock() | |||
| 267 | + | return a.CurrentRunTurns | |||
| 268 | + | } | |||
| 269 | + | ||||
| 270 | + | func (a *Agent) setTurnCounts(lifetime, currentRun int) { | |||
| 271 | + | if lifetime < 0 { | |||
| 272 | + | lifetime = 0 | |||
| 273 | + | } | |||
| 274 | + | if currentRun < 0 { | |||
| 275 | + | currentRun = 0 | |||
| 276 | + | } | |||
| 277 | + | a.lifecycleMu.Lock() | |||
| 278 | + | a.LifetimeTurns = lifetime | |||
| 279 | + | a.CurrentRunTurns = currentRun | |||
| 280 | + | a.updatedAt = time.Now() | |||
| 281 | + | a.lifecycleMu.Unlock() | |||
| 282 | + | } | |||
| 283 | + | ||||
| 284 | + | func (a *Agent) incrementTurnCounts() (lifetime, currentRun int) { | |||
| 285 | + | a.lifecycleMu.Lock() | |||
| 286 | + | a.LifetimeTurns++ | |||
| 287 | + | a.CurrentRunTurns++ | |||
| 288 | + | a.updatedAt = time.Now() | |||
| 289 | + | lifetime = a.LifetimeTurns | |||
| 290 | + | currentRun = a.CurrentRunTurns | |||
| 291 | + | a.lifecycleMu.Unlock() | |||
| 292 | + | return lifetime, currentRun | |||
| 293 | + | } | |||
| 294 | + | ||||
| 295 | + | func (a *Agent) resetCurrentRunTurns() (previous int) { | |||
| 296 | + | a.lifecycleMu.Lock() | |||
| 297 | + | previous = a.CurrentRunTurns | |||
| 298 | + | a.CurrentRunTurns = 0 | |||
| 299 | + | a.updatedAt = time.Now() | |||
| 300 | + | a.lifecycleMu.Unlock() | |||
| 301 | + | return previous | |||
| 302 | + | } | |||
| 303 | + | ||||
| 304 | + | func (a *Agent) setCurrentRunTurns(currentRun int) { | |||
| 305 | + | if currentRun < 0 { | |||
| 306 | + | currentRun = 0 | |||
| 307 | + | } | |||
| 308 | + | a.lifecycleMu.Lock() | |||
| 309 | + | a.CurrentRunTurns = currentRun | |||
| 310 | + | a.updatedAt = time.Now() | |||
| 311 | + | a.lifecycleMu.Unlock() | |||
| 312 | + | } | |||
| 313 | + | ||||
| 233 | // ProcessPIDValue returns the current worker pid, when known. | 314 | // ProcessPIDValue returns the current worker pid, when known. | ||
| 234 | func (a *Agent) ProcessPIDValue() int { | 315 | func (a *Agent) ProcessPIDValue() int { | ||
| 235 | a.lifecycleMu.Lock() | 316 | a.lifecycleMu.Lock() | ||
| @@ -406,9 +487,16 @@ func (a *Agent) setResumePrompt(prompt string, acceptedAt time.Time) (string, ti | |||||
| 406 | previousPrompt, previousAcceptedAt := a.resumePromptText, a.resumePromptAt | 487 | previousPrompt, previousAcceptedAt := a.resumePromptText, a.resumePromptAt | ||
| 407 | a.resumePromptText = prompt | 488 | a.resumePromptText = prompt | ||
| 408 | a.resumePromptAt = acceptedAt | 489 | a.resumePromptAt = acceptedAt | ||
| 490 | + | a.resumePromptID = uuid.NewString() | |||
| 409 | return previousPrompt, previousAcceptedAt | 491 | return previousPrompt, previousAcceptedAt | ||
| 410 | } | 492 | } | ||
| 411 | 493 | ||||
| 494 | + | func (a *Agent) resumePromptCommandID() string { | |||
| 495 | + | a.mu.Lock() | |||
| 496 | + | defer a.mu.Unlock() | |||
| 497 | + | return a.resumePromptID | |||
| 498 | + | } | |||
| 499 | + | ||||
| 412 | func (a *Agent) clearResumePrompt() bool { | 500 | func (a *Agent) clearResumePrompt() bool { | ||
| 413 | a.mu.Lock() | 501 | a.mu.Lock() | ||
| 414 | defer a.mu.Unlock() | 502 | defer a.mu.Unlock() | ||
| @@ -417,9 +505,106 @@ func (a *Agent) clearResumePrompt() bool { | |||||
| 417 | } | 505 | } | ||
| 418 | a.resumePromptText = "" | 506 | a.resumePromptText = "" | ||
| 419 | a.resumePromptAt = time.Time{} | 507 | a.resumePromptAt = time.Time{} | ||
| 508 | + | a.resumePromptID = "" | |||
| 420 | return true | 509 | return true | ||
| 421 | } | 510 | } | ||
| 422 | 511 | ||||
| 512 | + | func (a *Agent) queueResumePrompt(prompt string, acceptedAt time.Time) { | |||
| 513 | + | a.mu.Lock() | |||
| 514 | + | a.resumePromptQueue = append(a.resumePromptQueue, queuedResumePrompt{Prompt: prompt, AcceptedAt: acceptedAt, CommandID: uuid.NewString()}) | |||
| 515 | + | a.mu.Unlock() | |||
| 516 | + | } | |||
| 517 | + | ||||
| 518 | + | func (a *Agent) removeQueuedResumePrompt(prompt string, acceptedAt time.Time) bool { | |||
| 519 | + | a.mu.Lock() | |||
| 520 | + | defer a.mu.Unlock() | |||
| 521 | + | for i := len(a.resumePromptQueue) - 1; i >= 0; i-- { | |||
| 522 | + | queued := a.resumePromptQueue[i] | |||
| 523 | + | if queued.Prompt == prompt && queued.AcceptedAt.Equal(acceptedAt) { | |||
| 524 | + | a.resumePromptQueue = append(a.resumePromptQueue[:i], a.resumePromptQueue[i+1:]...) | |||
| 525 | + | return true | |||
| 526 | + | } | |||
| 527 | + | } | |||
| 528 | + | return false | |||
| 529 | + | } | |||
| 530 | + | ||||
| 531 | + | func (a *Agent) resumePromptQueueSnapshot() []queuedResumePrompt { | |||
| 532 | + | a.mu.Lock() | |||
| 533 | + | defer a.mu.Unlock() | |||
| 534 | + | return append([]queuedResumePrompt(nil), a.resumePromptQueue...) | |||
| 535 | + | } | |||
| 536 | + | ||||
| 537 | + | func (a *Agent) hasQueuedResumePrompt() bool { | |||
| 538 | + | a.mu.Lock() | |||
| 539 | + | defer a.mu.Unlock() | |||
| 540 | + | return len(a.resumePromptQueue) != 0 | |||
| 541 | + | } | |||
| 542 | + | ||||
| 543 | + | // startQueuedResumePrompt moves the oldest active follow-up into the | |||
| 544 | + | // in-flight prompt slot. The caller must persist the returned state before | |||
| 545 | + | // delivering the command so a host exit cannot lose the prompt. | |||
| 546 | + | func (a *Agent) startQueuedResumePrompt() (string, time.Time, string, bool) { | |||
| 547 | + | a.mu.Lock() | |||
| 548 | + | defer a.mu.Unlock() | |||
| 549 | + | if a.resumePromptText != "" || len(a.resumePromptQueue) == 0 { | |||
| 550 | + | return "", time.Time{}, "", false | |||
| 551 | + | } | |||
| 552 | + | queued := a.resumePromptQueue[0] | |||
| 553 | + | a.resumePromptQueue = a.resumePromptQueue[1:] | |||
| 554 | + | a.resumePromptText = queued.Prompt | |||
| 555 | + | a.resumePromptAt = queued.AcceptedAt | |||
| 556 | + | a.resumePromptID = queued.CommandID | |||
| 557 | + | if a.resumePromptID == "" { | |||
| 558 | + | a.resumePromptID = uuid.NewString() | |||
| 559 | + | } | |||
| 560 | + | return queued.Prompt, queued.AcceptedAt, a.resumePromptID, true | |||
| 561 | + | } | |||
| 562 | + | ||||
| 563 | + | func (a *Agent) restoreResumePromptToQueue(prompt string, acceptedAt time.Time) bool { | |||
| 564 | + | a.mu.Lock() | |||
| 565 | + | defer a.mu.Unlock() | |||
| 566 | + | if a.resumePromptText != prompt || !a.resumePromptAt.Equal(acceptedAt) { | |||
| 567 | + | return false | |||
| 568 | + | } | |||
| 569 | + | commandID := a.resumePromptID | |||
| 570 | + | if commandID == "" { | |||
| 571 | + | commandID = uuid.NewString() | |||
| 572 | + | } | |||
| 573 | + | a.resumePromptText = "" | |||
| 574 | + | a.resumePromptAt = time.Time{} | |||
| 575 | + | a.resumePromptID = "" | |||
| 576 | + | a.resumePromptQueue = append([]queuedResumePrompt{{Prompt: prompt, AcceptedAt: acceptedAt, CommandID: commandID}}, a.resumePromptQueue...) | |||
| 577 | + | return true | |||
| 578 | + | } | |||
| 579 | + | ||||
| 580 | + | func (a *Agent) rejectActiveResumePrompt() bool { | |||
| 581 | + | a.mu.Lock() | |||
| 582 | + | defer a.mu.Unlock() | |||
| 583 | + | if a.resumePromptText == "" { | |||
| 584 | + | return false | |||
| 585 | + | } | |||
| 586 | + | a.resumePromptQueue = append([]queuedResumePrompt{{Prompt: a.resumePromptText, AcceptedAt: a.resumePromptAt, CommandID: a.resumePromptID}}, a.resumePromptQueue...) | |||
| 587 | + | a.resumePromptText = "" | |||
| 588 | + | a.resumePromptAt = time.Time{} | |||
| 589 | + | a.resumePromptID = "" | |||
| 590 | + | return true | |||
| 591 | + | } | |||
| 592 | + | ||||
| 593 | + | func (a *Agent) setOnTurnIdle(fn func()) { | |||
| 594 | + | a.mu.Lock() | |||
| 595 | + | a.onTurnIdle = fn | |||
| 596 | + | a.mu.Unlock() | |||
| 597 | + | } | |||
| 598 | + | ||||
| 599 | + | func (a *Agent) notifyTurnIdle() { | |||
| 600 | + | a.mu.Lock() | |||
| 601 | + | fn := a.onTurnIdle | |||
| 602 | + | a.mu.Unlock() | |||
| 603 | + | if fn != nil { | |||
| 604 | + | fn() | |||
| 605 | + | } | |||
| 606 | + | } | |||
| 607 | + | ||||
| 423 | func (a *Agent) SetOnTurnEnd(fn func(step int, errMsg string)) { | 608 | func (a *Agent) SetOnTurnEnd(fn func(step int, errMsg string)) { | ||
| 424 | a.mu.Lock() | 609 | a.mu.Lock() | ||
| 425 | a.OnTurnEnd = fn | 610 | a.OnTurnEnd = fn | ||
| @@ -433,6 +618,18 @@ func (a *Agent) SetOnTurnEnd(fn func(step int, errMsg string)) { | |||||
| 433 | } | 618 | } | ||
| 434 | } | 619 | } | ||
| 435 | 620 | ||||
| 621 | + | func (a *Agent) recordPersistenceError(err error) { | |||
| 622 | + | if a == nil || err == nil { | |||
| 623 | + | return | |||
| 624 | + | } | |||
| 625 | + | a.mu.Lock() | |||
| 626 | + | a.lastErr = fmt.Errorf("subagent metadata persistence: %w", err) | |||
| 627 | + | if a.status != StatusDone && a.status != StatusFailed && a.status != StatusKilled { | |||
| 628 | + | a.activity = "metadata error: " + truncate(err.Error(), 120) | |||
| 629 | + | } | |||
| 630 | + | a.mu.Unlock() | |||
| 631 | + | } | |||
| 632 | + | ||||
| 436 | func (a *Agent) setActivity(msg string) { | 633 | func (a *Agent) setActivity(msg string) { | ||
| 437 | a.mu.Lock() | 634 | a.mu.Lock() | ||
| 438 | a.activity = strings.TrimSpace(msg) | 635 | a.activity = strings.TrimSpace(msg) | ||
M packages/agent/subagents/inbox.go +29 -7
| @@ -112,9 +112,12 @@ func (b *Inbox) sendBytesContext(ctx context.Context, data []byte) error { | |||
| 112 | 112 | } | |
| 113 | 113 | defer conn.SetWriteDeadline(time.Time{}) | |
| 114 | 114 | } | |
| 115 | - | if _, err := conn.Write(data); err != nil { | |
| 116 | - | // Drop the connection so the next call redials. The previous error is | |
| 117 | - | // more informative than the redial's would be, so surface this one. | |
| 115 | + | n, writeErr := conn.Write(data) | |
| 116 | + | if writeErr != nil { | |
| 117 | + | // Drop the connection so the next call redials. Once any bytes have | |
| 118 | + | // crossed the socket boundary, the outcome is ambiguous; returning the | |
| 119 | + | // caller's context error would invite a duplicate command with a new | |
| 120 | + | // identity. | |
| 118 | 121 | stopCancel() | |
| 119 | 122 | b.mu.Lock() | |
| 120 | 123 | if b.conn == conn { | |
| @@ -122,20 +125,33 @@ func (b *Inbox) sendBytesContext(ctx context.Context, data []byte) error { | |||
| 122 | 125 | b.conn = nil | |
| 123 | 126 | } | |
| 124 | 127 | b.mu.Unlock() | |
| 128 | + | if n > 0 { | |
| 129 | + | return fmt.Errorf("%w: wrote %d of %d bytes: %v", ErrDeliveryUnknown, n, len(data), writeErr) | |
| 130 | + | } | |
| 125 | 131 | if ctxErr := ctx.Err(); ctxErr != nil { | |
| 126 | 132 | return ctxErr | |
| 127 | 133 | } | |
| 128 | - | return err | |
| 134 | + | return writeErr | |
| 135 | + | } | |
| 136 | + | if n != len(data) { | |
| 137 | + | stopCancel() | |
| 138 | + | b.mu.Lock() | |
| 139 | + | if b.conn == conn { | |
| 140 | + | _ = b.conn.Close() | |
| 141 | + | b.conn = nil | |
| 142 | + | } | |
| 143 | + | b.mu.Unlock() | |
| 144 | + | return fmt.Errorf("%w: wrote %d of %d bytes", ErrDeliveryUnknown, n, len(data)) | |
| 129 | 145 | } | |
| 130 | 146 | if !stopCancel() { | |
| 131 | - | // Cancellation closed this connection after the write completed. Drop it | |
| 132 | - | // so the next command redials instead of reusing a closed stream. | |
| 147 | + | // Cancellation closed this connection after the complete command was | |
| 148 | + | // written. The worker can parse the newline-delimited command, so the | |
| 149 | + | // delivery succeeded even though the context is now canceled. | |
| 133 | 150 | b.mu.Lock() | |
| 134 | 151 | if b.conn == conn { | |
| 135 | 152 | b.conn = nil | |
| 136 | 153 | } | |
| 137 | 154 | b.mu.Unlock() | |
| 138 | - | return ctx.Err() | |
| 139 | 155 | } | |
| 140 | 156 | return nil | |
| 141 | 157 | } | |
| @@ -179,6 +195,12 @@ func (b *Inbox) Close() error { | |||
| 179 | 195 | // the TUI surfaces it as "agent <id> not listening yet". | |
| 180 | 196 | var ErrNotReady = errors.New("subagent: agent inbox not ready") | |
| 181 | 197 | ||
| 198 | + | // ErrDeliveryUnknown means a command write made progress but did not finish. | |
| 199 | + | // Callers must not replace the command with a fresh identity: the worker may | |
| 200 | + | // have received enough bytes to parse it, so retries must reuse the same | |
| 201 | + | // message id and be handled idempotently. | |
| 202 | + | var ErrDeliveryUnknown = errors.New("subagent: command delivery unknown") | |
| 203 | + | ||
| 182 | 204 | // dialUnixContext retries a Unix-socket connection briefly so the happy path | |
| 183 | 205 | // "child just started, supervisor sends right away" works without forcing the | |
| 184 | 206 | // caller to poll. Errors collapse onto ErrNotReady whenever the failure means | |
| @@ -112,9 +112,12 @@ func (b *Inbox) sendBytesContext(ctx context.Context, data []byte) error { | |||||
| 112 | } | 112 | } | ||
| 113 | defer conn.SetWriteDeadline(time.Time{}) | 113 | defer conn.SetWriteDeadline(time.Time{}) | ||
| 114 | } | 114 | } | ||
| 115 | - | if _, err := conn.Write(data); err != nil { | 115 | + | n, writeErr := conn.Write(data) |
| 116 | - | // Drop the connection so the next call redials. The previous error is | 116 | + | if writeErr != nil { |
| 117 | - | // more informative than the redial's would be, so surface this one. | 117 | + | // Drop the connection so the next call redials. Once any bytes have |
| 118 | + | // crossed the socket boundary, the outcome is ambiguous; returning the | |||
| 119 | + | // caller's context error would invite a duplicate command with a new | |||
| 120 | + | // identity. | |||
| 118 | stopCancel() | 121 | stopCancel() | ||
| 119 | b.mu.Lock() | 122 | b.mu.Lock() | ||
| 120 | if b.conn == conn { | 123 | if b.conn == conn { | ||
| @@ -122,20 +125,33 @@ func (b *Inbox) sendBytesContext(ctx context.Context, data []byte) error { | |||||
| 122 | b.conn = nil | 125 | b.conn = nil | ||
| 123 | } | 126 | } | ||
| 124 | b.mu.Unlock() | 127 | b.mu.Unlock() | ||
| 128 | + | if n > 0 { | |||
| 129 | + | return fmt.Errorf("%w: wrote %d of %d bytes: %v", ErrDeliveryUnknown, n, len(data), writeErr) | |||
| 130 | + | } | |||
| 125 | if ctxErr := ctx.Err(); ctxErr != nil { | 131 | if ctxErr := ctx.Err(); ctxErr != nil { | ||
| 126 | return ctxErr | 132 | return ctxErr | ||
| 127 | } | 133 | } | ||
| 128 | - | return err | 134 | + | return writeErr |
| 135 | + | } | |||
| 136 | + | if n != len(data) { | |||
| 137 | + | stopCancel() | |||
| 138 | + | b.mu.Lock() | |||
| 139 | + | if b.conn == conn { | |||
| 140 | + | _ = b.conn.Close() | |||
| 141 | + | b.conn = nil | |||
| 142 | + | } | |||
| 143 | + | b.mu.Unlock() | |||
| 144 | + | return fmt.Errorf("%w: wrote %d of %d bytes", ErrDeliveryUnknown, n, len(data)) | |||
| 129 | } | 145 | } | ||
| 130 | if !stopCancel() { | 146 | if !stopCancel() { | ||
| 131 | - | // Cancellation closed this connection after the write completed. Drop it | 147 | + | // Cancellation closed this connection after the complete command was |
| 132 | - | // so the next command redials instead of reusing a closed stream. | 148 | + | // written. The worker can parse the newline-delimited command, so the |
| 149 | + | // delivery succeeded even though the context is now canceled. | |||
| 133 | b.mu.Lock() | 150 | b.mu.Lock() | ||
| 134 | if b.conn == conn { | 151 | if b.conn == conn { | ||
| 135 | b.conn = nil | 152 | b.conn = nil | ||
| 136 | } | 153 | } | ||
| 137 | b.mu.Unlock() | 154 | b.mu.Unlock() | ||
| 138 | - | return ctx.Err() | |||
| 139 | } | 155 | } | ||
| 140 | return nil | 156 | return nil | ||
| 141 | } | 157 | } | ||
| @@ -179,6 +195,12 @@ func (b *Inbox) Close() error { | |||||
| 179 | // the TUI surfaces it as "agent <id> not listening yet". | 195 | // the TUI surfaces it as "agent <id> not listening yet". | ||
| 180 | var ErrNotReady = errors.New("subagent: agent inbox not ready") | 196 | var ErrNotReady = errors.New("subagent: agent inbox not ready") | ||
| 181 | 197 | ||||
| 198 | + | // ErrDeliveryUnknown means a command write made progress but did not finish. | |||
| 199 | + | // Callers must not replace the command with a fresh identity: the worker may | |||
| 200 | + | // have received enough bytes to parse it, so retries must reuse the same | |||
| 201 | + | // message id and be handled idempotently. | |||
| 202 | + | var ErrDeliveryUnknown = errors.New("subagent: command delivery unknown") | |||
| 203 | + | ||||
| 182 | // dialUnixContext retries a Unix-socket connection briefly so the happy path | 204 | // dialUnixContext retries a Unix-socket connection briefly so the happy path | ||
| 183 | // "child just started, supervisor sends right away" works without forcing the | 205 | // "child just started, supervisor sends right away" works without forcing the | ||
| 184 | // caller to poll. Errors collapse onto ErrNotReady whenever the failure means | 206 | // caller to poll. Errors collapse onto ErrNotReady whenever the failure means | ||
M packages/agent/subagents/inbox_test.go +23 -0
| @@ -127,6 +127,16 @@ func (c *blockingWriteConn) Close() error { | |||
| 127 | 127 | return c.Conn.Close() | |
| 128 | 128 | } | |
| 129 | 129 | ||
| 130 | + | type cancelAfterWriteConn struct { | |
| 131 | + | net.Conn | |
| 132 | + | cancel context.CancelFunc | |
| 133 | + | } | |
| 134 | + | ||
| 135 | + | func (c *cancelAfterWriteConn) Write(data []byte) (int, error) { | |
| 136 | + | c.cancel() | |
| 137 | + | return len(data), nil | |
| 138 | + | } | |
| 139 | + | ||
| 130 | 140 | type notifyingContext struct { | |
| 131 | 141 | context.Context | |
| 132 | 142 | doneCalled chan struct{} | |
| @@ -138,6 +148,19 @@ func (c *notifyingContext) Done() <-chan struct{} { | |||
| 138 | 148 | return c.Context.Done() | |
| 139 | 149 | } | |
| 140 | 150 | ||
| 151 | + | func TestInboxSendContextTreatsCompleteWriteBeforeCancellationAsDelivered(t *testing.T) { | |
| 152 | + | local, remote := net.Pipe() | |
| 153 | + | defer remote.Close() | |
| 154 | + | ctx, cancel := context.WithCancel(context.Background()) | |
| 155 | + | inbox := NewInbox("unused") | |
| 156 | + | inbox.conn = &cancelAfterWriteConn{Conn: local, cancel: cancel} | |
| 157 | + | defer inbox.Close() | |
| 158 | + | ||
| 159 | + | if err := inbox.sendBytesContext(ctx, []byte("follow-up\n")); err != nil { | |
| 160 | + | t.Fatalf("complete write after cancellation = %v, want success", err) | |
| 161 | + | } | |
| 162 | + | } | |
| 163 | + | ||
| 141 | 164 | func TestInboxSendContextCancelsBlockedWrite(t *testing.T) { | |
| 142 | 165 | local, remote := net.Pipe() | |
| 143 | 166 | defer remote.Close() | |
| @@ -127,6 +127,16 @@ func (c *blockingWriteConn) Close() error { | |||||
| 127 | return c.Conn.Close() | 127 | return c.Conn.Close() | ||
| 128 | } | 128 | } | ||
| 129 | 129 | ||||
| 130 | + | type cancelAfterWriteConn struct { | |||
| 131 | + | net.Conn | |||
| 132 | + | cancel context.CancelFunc | |||
| 133 | + | } | |||
| 134 | + | ||||
| 135 | + | func (c *cancelAfterWriteConn) Write(data []byte) (int, error) { | |||
| 136 | + | c.cancel() | |||
| 137 | + | return len(data), nil | |||
| 138 | + | } | |||
| 139 | + | ||||
| 130 | type notifyingContext struct { | 140 | type notifyingContext struct { | ||
| 131 | context.Context | 141 | context.Context | ||
| 132 | doneCalled chan struct{} | 142 | doneCalled chan struct{} | ||
| @@ -138,6 +148,19 @@ func (c *notifyingContext) Done() <-chan struct{} { | |||||
| 138 | return c.Context.Done() | 148 | return c.Context.Done() | ||
| 139 | } | 149 | } | ||
| 140 | 150 | ||||
| 151 | + | func TestInboxSendContextTreatsCompleteWriteBeforeCancellationAsDelivered(t *testing.T) { | |||
| 152 | + | local, remote := net.Pipe() | |||
| 153 | + | defer remote.Close() | |||
| 154 | + | ctx, cancel := context.WithCancel(context.Background()) | |||
| 155 | + | inbox := NewInbox("unused") | |||
| 156 | + | inbox.conn = &cancelAfterWriteConn{Conn: local, cancel: cancel} | |||
| 157 | + | defer inbox.Close() | |||
| 158 | + | ||||
| 159 | + | if err := inbox.sendBytesContext(ctx, []byte("follow-up\n")); err != nil { | |||
| 160 | + | t.Fatalf("complete write after cancellation = %v, want success", err) | |||
| 161 | + | } | |||
| 162 | + | } | |||
| 163 | + | ||||
| 141 | func TestInboxSendContextCancelsBlockedWrite(t *testing.T) { | 164 | func TestInboxSendContextCancelsBlockedWrite(t *testing.T) { | ||
| 142 | local, remote := net.Pipe() | 165 | local, remote := net.Pipe() | ||
| 143 | defer remote.Close() | 166 | defer remote.Close() | ||
M packages/agent/subagents/persist.go +282 -78
| @@ -54,6 +54,8 @@ type agentMeta struct { | |||
| 54 | 54 | WorkspaceBase string `json:"workspace_base,omitempty"` | |
| 55 | 55 | WorkspaceCapture CaptureMode `json:"workspace_capture,omitempty"` | |
| 56 | 56 | MaxTurns int `json:"max_turns,omitempty"` | |
| 57 | + | LifetimeTurns int `json:"lifetime_turns,omitempty"` | |
| 58 | + | CurrentRunTurns int `json:"current_run_turns,omitempty"` | |
| 57 | 59 | Timeout time.Duration `json:"timeout,omitempty"` | |
| 58 | 60 | Tools []string `json:"tools,omitempty"` | |
| 59 | 61 | WebSearchPolicy WebSearchPolicy `json:"web_search_policy,omitempty"` | |
| @@ -77,8 +79,10 @@ type agentMeta struct { | |||
| 77 | 79 | // ResumePrompt is a follow-up accepted by subagent_resume but not yet | |
| 78 | 80 | // acknowledged by the child with a turn.started event. It is durable so a | |
| 79 | 81 | // host exit while the worker is queued does not lose the manager signal. | |
| 80 | - | ResumePrompt string `json:"resume_prompt,omitempty"` | |
| 81 | - | ResumePromptAt time.Time `json:"resume_prompt_at,omitempty"` | |
| 82 | + | ResumePrompt string `json:"resume_prompt,omitempty"` | |
| 83 | + | ResumePromptAt time.Time `json:"resume_prompt_at,omitempty"` | |
| 84 | + | ResumePromptID string `json:"resume_prompt_id,omitempty"` | |
| 85 | + | ResumeQueue []queuedResumePrompt `json:"resume_queue,omitempty"` | |
| 82 | 86 | ||
| 83 | 87 | // SessionID, when non-empty, scopes the agent to a particular | |
| 84 | 88 | // host zut session: the dashboard only shows agents whose | |
| @@ -96,6 +100,12 @@ func metaPath(stateDir string) string { return filepath.Join(stateDir, "meta.jso | |||
| 96 | 100 | // write is atomic (tmp + rename) so a crash mid-write can't leave a | |
| 97 | 101 | // half-parsable file that fails Reload. | |
| 98 | 102 | func writeAgentMeta(stateDir string, a *Agent) error { | |
| 103 | + | if a == nil { | |
| 104 | + | return errors.New("subagent meta: nil agent") | |
| 105 | + | } | |
| 106 | + | a.persistMu.Lock() | |
| 107 | + | defer a.persistMu.Unlock() | |
| 108 | + | ||
| 99 | 109 | fastMode := a.FastMode | |
| 100 | 110 | webSearchPolicy := childWebSearchPolicy(a.WebSearchPolicy, a.Subagent, a.Tools) | |
| 101 | 111 | a.mu.Lock() | |
| @@ -104,6 +114,8 @@ func writeAgentMeta(stateDir string, a *Agent) error { | |||
| 104 | 114 | finished := a.finished | |
| 105 | 115 | resumePrompt := a.resumePromptText | |
| 106 | 116 | resumePromptAt := a.resumePromptAt | |
| 117 | + | resumePromptID := a.resumePromptID | |
| 118 | + | resumeQueue := append([]queuedResumePrompt(nil), a.resumePromptQueue...) | |
| 107 | 119 | errText := "" | |
| 108 | 120 | if a.lastErr != nil { | |
| 109 | 121 | errText = a.lastErr.Error() | |
| @@ -113,6 +125,8 @@ func writeAgentMeta(stateDir string, a *Agent) error { | |||
| 113 | 125 | processState := a.processState | |
| 114 | 126 | turnState := a.turnState | |
| 115 | 127 | currentTurnID := a.currentTurnID | |
| 128 | + | lifetimeTurns := a.LifetimeTurns | |
| 129 | + | currentRunTurns := a.CurrentRunTurns | |
| 116 | 130 | attempt := a.Attempt | |
| 117 | 131 | processPID := a.ProcessPID | |
| 118 | 132 | updatedAt := a.updatedAt | |
| @@ -143,6 +157,8 @@ func writeAgentMeta(stateDir string, a *Agent) error { | |||
| 143 | 157 | WorkspaceBase: a.WorkspaceBase, | |
| 144 | 158 | WorkspaceCapture: a.WorkspaceCapture, | |
| 145 | 159 | MaxTurns: a.MaxTurns, | |
| 160 | + | LifetimeTurns: lifetimeTurns, | |
| 161 | + | CurrentRunTurns: currentRunTurns, | |
| 146 | 162 | Timeout: a.Timeout, | |
| 147 | 163 | Tools: append([]string(nil), a.Tools...), | |
| 148 | 164 | WebSearchPolicy: webSearchPolicy, | |
| @@ -165,6 +181,8 @@ func writeAgentMeta(stateDir string, a *Agent) error { | |||
| 165 | 181 | SessionPath: a.SessionPath, | |
| 166 | 182 | ResumePrompt: resumePrompt, | |
| 167 | 183 | ResumePromptAt: resumePromptAt, | |
| 184 | + | ResumePromptID: resumePromptID, | |
| 185 | + | ResumeQueue: resumeQueue, | |
| 168 | 186 | SessionID: a.SessionID, | |
| 169 | 187 | } | |
| 170 | 188 | b, err := json.MarshalIndent(m, "", " ") | |
| @@ -175,14 +193,50 @@ func writeAgentMeta(stateDir string, a *Agent) error { | |||
| 175 | 193 | return fmt.Errorf("subagents meta dir: %w", err) | |
| 176 | 194 | } | |
| 177 | 195 | final := metaPath(stateDir) | |
| 178 | - | tmp := final + ".tmp" | |
| 179 | - | if err := os.WriteFile(tmp, append(b, '\n'), 0o600); err != nil { | |
| 196 | + | // Use a unique temporary file while holding the per-agent persistence | |
| 197 | + | // lock. A fixed meta.json.tmp is vulnerable to a concurrent writer (or a | |
| 198 | + | // stale file after a crash) replacing another writer's snapshot. | |
| 199 | + | tmpFile, err := os.CreateTemp(stateDir, ".meta.json.tmp-*") | |
| 200 | + | if err != nil { | |
| 201 | + | return fmt.Errorf("subagents meta temp: %w", err) | |
| 202 | + | } | |
| 203 | + | tmp := tmpFile.Name() | |
| 204 | + | removeTemp := true | |
| 205 | + | defer func() { | |
| 206 | + | _ = tmpFile.Close() | |
| 207 | + | if removeTemp { | |
| 208 | + | _ = os.Remove(tmp) | |
| 209 | + | } | |
| 210 | + | }() | |
| 211 | + | if err := tmpFile.Chmod(0o600); err != nil { | |
| 212 | + | return fmt.Errorf("subagents meta permissions: %w", err) | |
| 213 | + | } | |
| 214 | + | if _, err := tmpFile.Write(append(b, '\n')); err != nil { | |
| 180 | 215 | return fmt.Errorf("subagents meta write: %w", err) | |
| 181 | 216 | } | |
| 217 | + | if err := tmpFile.Sync(); err != nil { | |
| 218 | + | return fmt.Errorf("subagents meta sync: %w", err) | |
| 219 | + | } | |
| 220 | + | if err := tmpFile.Close(); err != nil { | |
| 221 | + | return fmt.Errorf("subagents meta close: %w", err) | |
| 222 | + | } | |
| 182 | 223 | if err := os.Rename(tmp, final); err != nil { | |
| 183 | - | _ = os.Remove(tmp) | |
| 184 | 224 | return fmt.Errorf("subagent meta rename: %w", err) | |
| 185 | 225 | } | |
| 226 | + | removeTemp = false | |
| 227 | + | // Sync the containing directory so the rename itself survives a host | |
| 228 | + | // crash, not just the bytes in the temporary file. | |
| 229 | + | dir, err := os.Open(stateDir) | |
| 230 | + | if err != nil { | |
| 231 | + | return fmt.Errorf("subagents meta directory open: %w", err) | |
| 232 | + | } | |
| 233 | + | if err := dir.Sync(); err != nil { | |
| 234 | + | _ = dir.Close() | |
| 235 | + | return fmt.Errorf("subagents meta directory sync: %w", err) | |
| 236 | + | } | |
| 237 | + | if err := dir.Close(); err != nil { | |
| 238 | + | return fmt.Errorf("subagents meta directory close: %w", err) | |
| 239 | + | } | |
| 186 | 240 | return nil | |
| 187 | 241 | } | |
| 188 | 242 | ||
| @@ -420,57 +474,61 @@ func (f *Supervisor) buildDetachedAgent(m agentMeta) *Agent { | |||
| 420 | 474 | agentStateDir = filepath.Dir(m.EventLogPath) | |
| 421 | 475 | } | |
| 422 | 476 | a := &Agent{ | |
| 423 | - | ID: m.ID, | |
| 424 | - | Task: m.Task, | |
| 425 | - | OriginalTask: firstNonEmpty(m.OriginalTask, m.Task), | |
| 426 | - | Dir: dir, | |
| 427 | - | RepositoryRoot: firstNonEmpty(m.RepositoryRoot, f.cfg.RepoRoot), | |
| 428 | - | Started: m.Started, | |
| 429 | - | ParentID: m.ParentID, | |
| 430 | - | BatchID: m.BatchID, | |
| 431 | - | RootSessionID: m.RootSessionID, | |
| 432 | - | Model: m.Model, | |
| 433 | - | Provider: m.Provider, | |
| 434 | - | BaseURL: m.BaseURL, | |
| 435 | - | InsecureTLS: m.InsecureTLS, | |
| 436 | - | Reasoning: m.Reasoning, | |
| 437 | - | FastMode: fastMode, | |
| 438 | - | Subagent: m.Subagent, | |
| 439 | - | WorkspaceMode: m.WorkspaceMode, | |
| 440 | - | WorkspacePath: m.WorkspacePath, | |
| 441 | - | WorkspaceBase: m.WorkspaceBase, | |
| 442 | - | WorkspaceCapture: m.WorkspaceCapture, | |
| 443 | - | MaxTurns: m.MaxTurns, | |
| 444 | - | Timeout: m.Timeout, | |
| 445 | - | Tools: append([]string(nil), m.Tools...), | |
| 446 | - | WebSearchPolicy: childWebSearchPolicy(m.WebSearchPolicy, m.Subagent, m.Tools), | |
| 447 | - | Attempt: m.Attempt, | |
| 448 | - | ProcessPID: m.ProcessPID, | |
| 449 | - | InboxPath: m.InboxPath, | |
| 450 | - | EventLogPath: m.EventLogPath, | |
| 451 | - | SessionPath: m.SessionPath, | |
| 452 | - | resumePromptText: m.ResumePrompt, | |
| 453 | - | resumePromptAt: m.ResumePromptAt, | |
| 454 | - | SessionID: m.SessionID, | |
| 455 | - | inbox: NewInbox(m.InboxPath), | |
| 456 | - | status: status, | |
| 457 | - | activity: activity, | |
| 458 | - | finished: m.Finished, | |
| 459 | - | lastErr: lastErr, | |
| 460 | - | legacyEventState: legacyEventState, | |
| 461 | - | processState: processState, | |
| 462 | - | turnState: turnState, | |
| 463 | - | currentTurnID: m.CurrentTurnID, | |
| 464 | - | updatedAt: m.UpdatedAt, | |
| 465 | - | lastActivity: lastActivity, | |
| 466 | - | stateDir: agentStateDir, | |
| 467 | - | resultRef: m.ResultRef, | |
| 468 | - | patchRef: m.PatchRef, | |
| 469 | - | changedFiles: append([]string(nil), m.ChangedFiles...), | |
| 470 | - | maxOutputBytes: f.cfg.Policy.MaxOutputBytes, | |
| 471 | - | maxOutputLines: f.cfg.Policy.MaxOutputLines, | |
| 472 | - | done: make(chan struct{}), | |
| 473 | - | turnResults: make(chan *TurnResult, 16), | |
| 477 | + | ID: m.ID, | |
| 478 | + | Task: m.Task, | |
| 479 | + | OriginalTask: firstNonEmpty(m.OriginalTask, m.Task), | |
| 480 | + | Dir: dir, | |
| 481 | + | RepositoryRoot: firstNonEmpty(m.RepositoryRoot, f.cfg.RepoRoot), | |
| 482 | + | Started: m.Started, | |
| 483 | + | ParentID: m.ParentID, | |
| 484 | + | BatchID: m.BatchID, | |
| 485 | + | RootSessionID: m.RootSessionID, | |
| 486 | + | Model: m.Model, | |
| 487 | + | Provider: m.Provider, | |
| 488 | + | BaseURL: m.BaseURL, | |
| 489 | + | InsecureTLS: m.InsecureTLS, | |
| 490 | + | Reasoning: m.Reasoning, | |
| 491 | + | FastMode: fastMode, | |
| 492 | + | Subagent: m.Subagent, | |
| 493 | + | WorkspaceMode: m.WorkspaceMode, | |
| 494 | + | WorkspacePath: m.WorkspacePath, | |
| 495 | + | WorkspaceBase: m.WorkspaceBase, | |
| 496 | + | WorkspaceCapture: m.WorkspaceCapture, | |
| 497 | + | MaxTurns: m.MaxTurns, | |
| 498 | + | LifetimeTurns: m.LifetimeTurns, | |
| 499 | + | CurrentRunTurns: m.CurrentRunTurns, | |
| 500 | + | Timeout: m.Timeout, | |
| 501 | + | Tools: append([]string(nil), m.Tools...), | |
| 502 | + | WebSearchPolicy: childWebSearchPolicy(m.WebSearchPolicy, m.Subagent, m.Tools), | |
| 503 | + | Attempt: m.Attempt, | |
| 504 | + | ProcessPID: m.ProcessPID, | |
| 505 | + | InboxPath: m.InboxPath, | |
| 506 | + | EventLogPath: m.EventLogPath, | |
| 507 | + | SessionPath: m.SessionPath, | |
| 508 | + | resumePromptText: m.ResumePrompt, | |
| 509 | + | resumePromptAt: m.ResumePromptAt, | |
| 510 | + | resumePromptID: m.ResumePromptID, | |
| 511 | + | resumePromptQueue: append([]queuedResumePrompt(nil), m.ResumeQueue...), | |
| 512 | + | SessionID: m.SessionID, | |
| 513 | + | inbox: NewInbox(m.InboxPath), | |
| 514 | + | status: status, | |
| 515 | + | activity: activity, | |
| 516 | + | finished: m.Finished, | |
| 517 | + | lastErr: lastErr, | |
| 518 | + | legacyEventState: legacyEventState, | |
| 519 | + | processState: processState, | |
| 520 | + | turnState: turnState, | |
| 521 | + | currentTurnID: m.CurrentTurnID, | |
| 522 | + | updatedAt: m.UpdatedAt, | |
| 523 | + | lastActivity: lastActivity, | |
| 524 | + | stateDir: agentStateDir, | |
| 525 | + | resultRef: m.ResultRef, | |
| 526 | + | patchRef: m.PatchRef, | |
| 527 | + | changedFiles: append([]string(nil), m.ChangedFiles...), | |
| 528 | + | maxOutputBytes: f.cfg.Policy.MaxOutputBytes, | |
| 529 | + | maxOutputLines: f.cfg.Policy.MaxOutputLines, | |
| 530 | + | done: make(chan struct{}), | |
| 531 | + | turnResults: make(chan *TurnResult, 16), | |
| 474 | 532 | } | |
| 475 | 533 | if a.updatedAt.IsZero() { | |
| 476 | 534 | a.updatedAt = lastActivity | |
| @@ -542,7 +600,7 @@ func (a *Agent) loadTranscript() error { | |||
| 542 | 600 | if err != nil { | |
| 543 | 601 | return fmt.Errorf("read subagent transcript: %w", err) | |
| 544 | 602 | } | |
| 545 | - | if prompt, acceptedAt := a.ResumePromptInfo(); prompt != "" && resumePromptAcknowledged(events, acceptedAt) { | |
| 603 | + | if prompt, acceptedAt := a.ResumePromptInfo(); prompt != "" && resumePromptAcknowledged(events, prompt, acceptedAt) { | |
| 546 | 604 | a.clearResumePrompt() | |
| 547 | 605 | } | |
| 548 | 606 | a.mu.Lock() | |
| @@ -583,12 +641,23 @@ func (a *Agent) loadTranscript() error { | |||
| 583 | 641 | // Status precedence: explicit lifecycle events (agent_stopped) win | |
| 584 | 642 | // over inferred ones (assistant_message → idle). If the log contains | |
| 585 | 643 | // no terminator we keep status=StatusDetached so the user can resume. | |
| 586 | - | func resumePromptAcknowledged(events []Event, acceptedAt time.Time) bool { | |
| 587 | - | if acceptedAt.IsZero() { | |
| 644 | + | func resumePromptAcknowledged(events []Event, prompt string, acceptedAt time.Time) bool { | |
| 645 | + | if prompt == "" || acceptedAt.IsZero() { | |
| 588 | 646 | return false | |
| 589 | 647 | } | |
| 648 | + | userDurable := false | |
| 590 | 649 | for _, ev := range events { | |
| 591 | - | if (ev.Type == EventTurnStarted || ev.Type == "turn_start") && !ev.Time.Before(acceptedAt) { | |
| 650 | + | if ev.Time.Before(acceptedAt) { | |
| 651 | + | continue | |
| 652 | + | } | |
| 653 | + | if ev.Type == "user_message" && userMessageText(ev) == prompt { | |
| 654 | + | // The worker emits this event only after the matching session | |
| 655 | + | // message has been appended and flushed. A crash here leaves the | |
| 656 | + | // prompt pending, which is intentionally safe to replay. | |
| 657 | + | userDurable = true | |
| 658 | + | continue | |
| 659 | + | } | |
| 660 | + | if userDurable && isDelegatedTurnStart(ev) { | |
| 592 | 661 | return true | |
| 593 | 662 | } | |
| 594 | 663 | } | |
| @@ -599,14 +668,11 @@ func replayEventsIntoAgent(a *Agent, evs []Event) { | |||
| 599 | 668 | terminal := false | |
| 600 | 669 | for _, ev := range evs { | |
| 601 | 670 | switch ev.Type { | |
| 602 | - | case EventTurnStarted, "turn_start": | |
| 603 | - | turnID := ev.TurnID | |
| 604 | - | if turnID == "" { | |
| 605 | - | if step, ok := ev.Data["step"].(float64); ok { | |
| 606 | - | turnID = fmt.Sprintf("turn-%d", int(step)) | |
| 607 | - | } | |
| 671 | + | case EventTurnStarted: | |
| 672 | + | if !isDelegatedTurnStart(ev) { | |
| 673 | + | continue | |
| 608 | 674 | } | |
| 609 | - | a.setTurnState(TurnRunning, turnID) | |
| 675 | + | a.setTurnState(TurnRunning, eventTurnID(ev)) | |
| 610 | 676 | case EventTurnResult, "turn_result": | |
| 611 | 677 | if result, err := decodeTurnResultEvent(ev, a.ID, a.maxOutputBytes, a.maxOutputLines); err == nil { | |
| 612 | 678 | a.setResult(result) | |
| @@ -735,9 +801,16 @@ func (f *Supervisor) ResumeSession(ctx context.Context, id string) (*Agent, erro | |||
| 735 | 801 | ||
| 736 | 802 | // ResumeWithPrompt continues an agent with a manager follow-up while retaining | |
| 737 | 803 | // its existing session. An idle live worker receives the turn through its | |
| 738 | - | // inbox; a terminal worker is restarted with the follow-up as its initial | |
| 739 | - | // prompt, avoiding a race with inbox listener startup. | |
| 804 | + | // inbox; an active live worker queues the follow-up for the next idle turn; a | |
| 805 | + | // terminal worker is restarted with the follow-up as its initial prompt, | |
| 806 | + | // avoiding a race with inbox listener startup. | |
| 740 | 807 | func (f *Supervisor) ResumeWithPrompt(ctx context.Context, id, prompt string) (*Agent, error) { | |
| 808 | + | if ctx == nil { | |
| 809 | + | ctx = context.Background() | |
| 810 | + | } | |
| 811 | + | if err := ctx.Err(); err != nil { | |
| 812 | + | return nil, err | |
| 813 | + | } | |
| 741 | 814 | if strings.TrimSpace(prompt) == "" { | |
| 742 | 815 | return nil, fmt.Errorf("subagent: follow-up prompt is required") | |
| 743 | 816 | } | |
| @@ -750,15 +823,42 @@ func (f *Supervisor) ResumeWithPrompt(ctx context.Context, id, prompt string) (* | |||
| 750 | 823 | } | |
| 751 | 824 | status := existing.Status() | |
| 752 | 825 | if status == StatusRunning { | |
| 753 | - | if existing.ProcessState() != ProcessAlive || existing.TurnState() != TurnIdle { | |
| 826 | + | if existing.ProcessState() != ProcessAlive { | |
| 754 | 827 | f.operationMu.Unlock() | |
| 755 | - | return nil, fmt.Errorf("subagent: agent %s is still working; wait for it to become idle before sending a follow-up", existing.ID) | |
| 828 | + | return nil, fmt.Errorf("subagent: agent %s is still working; wait for its worker to become available before sending a follow-up", existing.ID) | |
| 756 | 829 | } | |
| 757 | 830 | if existing.inbox == nil { | |
| 758 | 831 | f.operationMu.Unlock() | |
| 759 | 832 | return nil, fmt.Errorf("subagent: agent %s has no inbox", existing.ID) | |
| 760 | 833 | } | |
| 834 | + | if err := ctx.Err(); err != nil { | |
| 835 | + | f.operationMu.Unlock() | |
| 836 | + | return nil, err | |
| 837 | + | } | |
| 838 | + | if existing.TurnState() != TurnRunning && existing.hasQueuedResumePrompt() { | |
| 839 | + | f.operationMu.Unlock() | |
| 840 | + | return nil, fmt.Errorf("subagent: agent %s already has a follow-up pending delivery", existing.ID) | |
| 841 | + | } | |
| 842 | + | if existing.TurnState() == TurnRunning { | |
| 843 | + | // Active turns cannot accept a command through the worker inbox yet. | |
| 844 | + | // Keep the prompt durable and let the idle transition deliver it in | |
| 845 | + | // order after the current message turn finishes. | |
| 846 | + | acceptedAt := f.cfg.Now() | |
| 847 | + | existing.queueResumePrompt(prompt, acceptedAt) | |
| 848 | + | if err := f.persistAgent(existing); err != nil { | |
| 849 | + | existing.removeQueuedResumePrompt(prompt, acceptedAt) | |
| 850 | + | f.operationMu.Unlock() | |
| 851 | + | return nil, fmt.Errorf("subagent: persist queued follow-up for %s: %w", existing.ID, err) | |
| 852 | + | } | |
| 853 | + | f.operationMu.Unlock() | |
| 854 | + | return existing, nil | |
| 855 | + | } | |
| 856 | + | if existing.TurnState() != TurnIdle { | |
| 857 | + | f.operationMu.Unlock() | |
| 858 | + | return nil, fmt.Errorf("subagent: agent %s is still working; wait for it to become idle before sending a follow-up", existing.ID) | |
| 859 | + | } | |
| 761 | 860 | turnID := existing.CurrentTurnID() | |
| 861 | + | previousRunTurns := existing.resetCurrentRunTurns() | |
| 762 | 862 | // Reserve the worker before writing to its inbox. A second manager call | |
| 763 | 863 | // must not enqueue another turn in the small interval before the worker | |
| 764 | 864 | // reports its turn.start event. Persist the prompt first so a host exit | |
| @@ -766,18 +866,39 @@ func (f *Supervisor) ResumeWithPrompt(ctx context.Context, id, prompt string) (* | |||
| 766 | 866 | previousPrompt, previousPromptAt := existing.setResumePrompt(prompt, f.cfg.Now()) | |
| 767 | 867 | existing.setTurnState(TurnQueued, turnID) | |
| 768 | 868 | inbox := existing.inbox | |
| 769 | - | command := NewCommand(CommandTurnStart, existing.ID, turnID, TurnStartPayload{Prompt: prompt}) | |
| 770 | - | f.persistAgent(existing) | |
| 869 | + | command := NewCommand(CommandTurnStart, existing.ID, turnID, TurnStartPayload{Prompt: prompt, NewRun: true}) | |
| 870 | + | command.MessageID = existing.resumePromptCommandID() | |
| 871 | + | if err := f.persistAgent(existing); err != nil { | |
| 872 | + | existing.setTurnState(TurnIdle, turnID) | |
| 873 | + | existing.setCurrentRunTurns(previousRunTurns) | |
| 874 | + | existing.setResumePrompt(previousPrompt, previousPromptAt) | |
| 875 | + | f.operationMu.Unlock() | |
| 876 | + | return nil, fmt.Errorf("subagent: persist follow-up for %s: %w", existing.ID, err) | |
| 877 | + | } | |
| 771 | 878 | f.operationMu.Unlock() | |
| 772 | 879 | ||
| 773 | 880 | if err := inbox.SendCommandContext(ctx, command); err != nil { | |
| 881 | + | retryUnknown := false | |
| 774 | 882 | f.operationMu.Lock() | |
| 775 | 883 | if existing.Status() == StatusRunning && existing.TurnState() == TurnQueued && existing.CurrentTurnID() == turnID { | |
| 776 | 884 | existing.setTurnState(TurnIdle, turnID) | |
| 777 | - | existing.setResumePrompt(previousPrompt, previousPromptAt) | |
| 778 | - | f.persistAgent(existing) | |
| 885 | + | existing.setCurrentRunTurns(previousRunTurns) | |
| 886 | + | if errors.Is(err, ErrDeliveryUnknown) { | |
| 887 | + | existing.rejectActiveResumePrompt() | |
| 888 | + | retryUnknown = true | |
| 889 | + | } else { | |
| 890 | + | existing.setResumePrompt(previousPrompt, previousPromptAt) | |
| 891 | + | } | |
| 892 | + | if persistErr := f.persistAgent(existing); persistErr != nil { | |
| 893 | + | err = fmt.Errorf("%w; persist recovery: %v", err, persistErr) | |
| 894 | + | } | |
| 779 | 895 | } | |
| 780 | 896 | f.operationMu.Unlock() | |
| 897 | + | if retryUnknown { | |
| 898 | + | // The command keeps its durable identity. A retry is safe whether | |
| 899 | + | // the worker parsed the first write or only received a prefix. | |
| 900 | + | f.dispatchQueuedResume(existing) | |
| 901 | + | } | |
| 781 | 902 | return nil, fmt.Errorf("subagent: send follow-up to %s: %w", existing.ID, err) | |
| 782 | 903 | } | |
| 783 | 904 | return existing, nil | |
| @@ -787,6 +908,57 @@ func (f *Supervisor) ResumeWithPrompt(ctx context.Context, id, prompt string) (* | |||
| 787 | 908 | return f.resume(ctx, id, true, prompt) | |
| 788 | 909 | } | |
| 789 | 910 | ||
| 911 | + | // dispatchQueuedResume delivers the oldest follow-up accepted during an | |
| 912 | + | // active turn after the worker reports that its inbox can accept a new turn. | |
| 913 | + | // The operation lock serializes this transition with manager resume calls and | |
| 914 | + | // the agent's durable prompt state prevents a host exit from losing it. | |
| 915 | + | func (f *Supervisor) dispatchQueuedResume(a *Agent) { | |
| 916 | + | if a == nil { | |
| 917 | + | return | |
| 918 | + | } | |
| 919 | + | f.operationMu.Lock() | |
| 920 | + | if a.Status() != StatusRunning || a.ProcessState() != ProcessAlive || a.TurnState() != TurnIdle || a.inbox == nil { | |
| 921 | + | f.operationMu.Unlock() | |
| 922 | + | return | |
| 923 | + | } | |
| 924 | + | prompt, acceptedAt, commandID, ok := a.startQueuedResumePrompt() | |
| 925 | + | if !ok { | |
| 926 | + | f.operationMu.Unlock() | |
| 927 | + | return | |
| 928 | + | } | |
| 929 | + | turnID := a.CurrentTurnID() | |
| 930 | + | previousRunTurns := a.resetCurrentRunTurns() | |
| 931 | + | a.setTurnState(TurnQueued, turnID) | |
| 932 | + | inbox := a.inbox | |
| 933 | + | command := NewCommand(CommandTurnStart, a.ID, turnID, TurnStartPayload{Prompt: prompt, NewRun: true}) | |
| 934 | + | command.MessageID = commandID | |
| 935 | + | if err := f.persistAgent(a); err != nil { | |
| 936 | + | a.restoreResumePromptToQueue(prompt, acceptedAt) | |
| 937 | + | a.setCurrentRunTurns(previousRunTurns) | |
| 938 | + | a.setTurnState(TurnIdle, turnID) | |
| 939 | + | f.operationMu.Unlock() | |
| 940 | + | return | |
| 941 | + | } | |
| 942 | + | f.operationMu.Unlock() | |
| 943 | + | ||
| 944 | + | if err := inbox.SendCommand(command); err == nil { | |
| 945 | + | return | |
| 946 | + | } | |
| 947 | + | ||
| 948 | + | // A failed delivery must put the prompt back at the head of the queue. | |
| 949 | + | // Keep the worker idle so a later retry, or an explicit resume, can send | |
| 950 | + | // it without skipping any subsequent follow-ups. | |
| 951 | + | f.operationMu.Lock() | |
| 952 | + | if a.Status() == StatusRunning && a.ProcessState() == ProcessAlive && a.TurnState() == TurnQueued && a.CurrentTurnID() == turnID { | |
| 953 | + | if a.restoreResumePromptToQueue(prompt, acceptedAt) { | |
| 954 | + | a.setCurrentRunTurns(previousRunTurns) | |
| 955 | + | a.setTurnState(TurnIdle, turnID) | |
| 956 | + | f.persistAgent(a) | |
| 957 | + | } | |
| 958 | + | } | |
| 959 | + | f.operationMu.Unlock() | |
| 960 | + | } | |
| 961 | + | ||
| 790 | 962 | // RestartTask intentionally runs the stored original task again in a fresh | |
| 791 | 963 | // worker attempt. Callers should use ResumeSession for normal recovery. | |
| 792 | 964 | func (f *Supervisor) RestartTask(ctx context.Context, id string) (*Agent, error) { | |
| @@ -852,10 +1024,35 @@ func (f *Supervisor) resume(ctx context.Context, id string, resuming bool, resum | |||
| 852 | 1024 | } | |
| 853 | 1025 | now := f.cfg.Now() | |
| 854 | 1026 | fastMode := existing.FastMode | |
| 1027 | + | lifetimeTurns := existing.LifetimeTurnsValue() | |
| 1028 | + | currentRunTurns := existing.CurrentRunTurnsValue() | |
| 855 | 1029 | pendingResumePrompt, pendingResumePromptAt := existing.ResumePromptInfo() | |
| 1030 | + | pendingResumeQueue := existing.resumePromptQueueSnapshot() | |
| 856 | 1031 | if resumePrompt != "" { | |
| 857 | - | pendingResumePrompt = resumePrompt | |
| 858 | - | pendingResumePromptAt = now | |
| 1032 | + | currentRunTurns = 0 | |
| 1033 | + | if pendingResumePrompt == "" { | |
| 1034 | + | pendingResumePrompt = resumePrompt | |
| 1035 | + | pendingResumePromptAt = now | |
| 1036 | + | } else { | |
| 1037 | + | pendingResumeQueue = append(pendingResumeQueue, queuedResumePrompt{Prompt: resumePrompt, AcceptedAt: now}) | |
| 1038 | + | } | |
| 1039 | + | } | |
| 1040 | + | if resuming && resumePrompt == "" && pendingResumePrompt == "" && len(pendingResumeQueue) > 0 { | |
| 1041 | + | // A host can exit after accepting an active-worker follow-up but before | |
| 1042 | + | // dispatching it. Make the first durable queued prompt the resumed | |
| 1043 | + | // worker's initial turn rather than leaving the queue stranded. | |
| 1044 | + | pendingResumePrompt = pendingResumeQueue[0].Prompt | |
| 1045 | + | pendingResumePromptAt = pendingResumeQueue[0].AcceptedAt | |
| 1046 | + | pendingResumeQueue = pendingResumeQueue[1:] | |
| 1047 | + | } | |
| 1048 | + | if resuming && pendingResumePrompt != "" { | |
| 1049 | + | // A retained follow-up is an explicit new run, including a queued | |
| 1050 | + | // follow-up recovered after a host restart. Preserve lifetime turns but | |
| 1051 | + | // never carry the previous run's allowance into this one. | |
| 1052 | + | currentRunTurns = 0 | |
| 1053 | + | } | |
| 1054 | + | if !resuming { | |
| 1055 | + | currentRunTurns = 0 | |
| 859 | 1056 | } | |
| 860 | 1057 | m := agentMeta{ | |
| 861 | 1058 | ID: existing.ID, Task: existing.Task, | |
| @@ -868,7 +1065,7 @@ func (f *Supervisor) resume(ctx context.Context, id string, resuming bool, resum | |||
| 868 | 1065 | Subagent: existing.Subagent, | |
| 869 | 1066 | WorkspaceMode: existing.WorkspaceMode, WorkspacePath: existing.WorkspacePath, | |
| 870 | 1067 | WorkspaceBase: existing.WorkspaceBase, WorkspaceCapture: existing.WorkspaceCapture, | |
| 871 | - | MaxTurns: existing.MaxTurns, Timeout: existing.Timeout, Tools: append([]string(nil), existing.Tools...), | |
| 1068 | + | MaxTurns: existing.MaxTurns, LifetimeTurns: lifetimeTurns, CurrentRunTurns: currentRunTurns, Timeout: existing.Timeout, Tools: append([]string(nil), existing.Tools...), | |
| 872 | 1069 | WebSearchPolicy: childWebSearchPolicy(existing.WebSearchPolicy, existing.Subagent, existing.Tools), | |
| 873 | 1070 | CurrentTurnID: existingSnapshot.CurrentTurnID, Attempt: existingSnapshot.Attempt, | |
| 874 | 1071 | SessionID: existing.SessionID, | |
| @@ -876,6 +1073,8 @@ func (f *Supervisor) resume(ctx context.Context, id string, resuming bool, resum | |||
| 876 | 1073 | SessionPath: existing.SessionPath, | |
| 877 | 1074 | ResumePrompt: pendingResumePrompt, | |
| 878 | 1075 | ResumePromptAt: pendingResumePromptAt, | |
| 1076 | + | ResumePromptID: existing.resumePromptCommandID(), | |
| 1077 | + | ResumeQueue: pendingResumeQueue, | |
| 879 | 1078 | } | |
| 880 | 1079 | ||
| 881 | 1080 | if len(m.Tools) == 0 && len(f.cfg.Policy.AllowedTools) > 0 { | |
| @@ -954,6 +1153,8 @@ func (f *Supervisor) resume(ctx context.Context, id string, resuming bool, resum | |||
| 954 | 1153 | WorkspaceBase: m.WorkspaceBase, | |
| 955 | 1154 | WorkspaceCapture: m.WorkspaceCapture, | |
| 956 | 1155 | MaxTurns: maxTurns, | |
| 1156 | + | LifetimeTurns: m.LifetimeTurns, | |
| 1157 | + | CurrentRunTurns: m.CurrentRunTurns, | |
| 957 | 1158 | Timeout: timeout, | |
| 958 | 1159 | Attempt: m.Attempt, | |
| 959 | 1160 | HeartbeatInterval: f.cfg.Policy.HeartbeatInterval, | |
| @@ -965,6 +1166,8 @@ func (f *Supervisor) resume(ctx context.Context, id string, resuming bool, resum | |||
| 965 | 1166 | Resuming: resuming, | |
| 966 | 1167 | resumePromptText: pendingResumePrompt, | |
| 967 | 1168 | resumePromptAt: pendingResumePromptAt, | |
| 1169 | + | resumePromptID: m.ResumePromptID, | |
| 1170 | + | resumePromptQueue: append([]queuedResumePrompt(nil), pendingResumeQueue...), | |
| 968 | 1171 | inbox: NewInbox(m.InboxPath), | |
| 969 | 1172 | status: StatusPending, | |
| 970 | 1173 | activity: "resuming", | |
| @@ -989,6 +1192,7 @@ func (f *Supervisor) resume(ctx context.Context, id string, resuming bool, resum | |||
| 989 | 1192 | } | |
| 990 | 1193 | a.ctx, a.cancel = runCtx, cancel | |
| 991 | 1194 | a.persistFn = f.persistAgent | |
| 1195 | + | a.setOnTurnIdle(func() { f.dispatchQueuedResume(a) }) | |
| 992 | 1196 | a.workspaceCleanup = func() error { return workspace.Cleanup(context.Background()) } | |
| 993 | 1197 | a.workspaceCapture = func() (WorkspaceCapture, error) { return workspace.Capture(context.Background()) } | |
| 994 | 1198 | a.runner = f.cfg.NewRunner(a) | |
| @@ -54,6 +54,8 @@ type agentMeta struct { | |||||
| 54 | WorkspaceBase string `json:"workspace_base,omitempty"` | 54 | WorkspaceBase string `json:"workspace_base,omitempty"` | ||
| 55 | WorkspaceCapture CaptureMode `json:"workspace_capture,omitempty"` | 55 | WorkspaceCapture CaptureMode `json:"workspace_capture,omitempty"` | ||
| 56 | MaxTurns int `json:"max_turns,omitempty"` | 56 | MaxTurns int `json:"max_turns,omitempty"` | ||
| 57 | + | LifetimeTurns int `json:"lifetime_turns,omitempty"` | |||
| 58 | + | CurrentRunTurns int `json:"current_run_turns,omitempty"` | |||
| 57 | Timeout time.Duration `json:"timeout,omitempty"` | 59 | Timeout time.Duration `json:"timeout,omitempty"` | ||
| 58 | Tools []string `json:"tools,omitempty"` | 60 | Tools []string `json:"tools,omitempty"` | ||
| 59 | WebSearchPolicy WebSearchPolicy `json:"web_search_policy,omitempty"` | 61 | WebSearchPolicy WebSearchPolicy `json:"web_search_policy,omitempty"` | ||
| @@ -77,8 +79,10 @@ type agentMeta struct { | |||||
| 77 | // ResumePrompt is a follow-up accepted by subagent_resume but not yet | 79 | // ResumePrompt is a follow-up accepted by subagent_resume but not yet | ||
| 78 | // acknowledged by the child with a turn.started event. It is durable so a | 80 | // acknowledged by the child with a turn.started event. It is durable so a | ||
| 79 | // host exit while the worker is queued does not lose the manager signal. | 81 | // host exit while the worker is queued does not lose the manager signal. | ||
| 80 | - | ResumePrompt string `json:"resume_prompt,omitempty"` | 82 | + | ResumePrompt string `json:"resume_prompt,omitempty"` |
| 81 | - | ResumePromptAt time.Time `json:"resume_prompt_at,omitempty"` | 83 | + | ResumePromptAt time.Time `json:"resume_prompt_at,omitempty"` |
| 84 | + | ResumePromptID string `json:"resume_prompt_id,omitempty"` | |||
| 85 | + | ResumeQueue []queuedResumePrompt `json:"resume_queue,omitempty"` | |||
| 82 | 86 | ||||
| 83 | // SessionID, when non-empty, scopes the agent to a particular | 87 | // SessionID, when non-empty, scopes the agent to a particular | ||
| 84 | // host zut session: the dashboard only shows agents whose | 88 | // host zut session: the dashboard only shows agents whose | ||
| @@ -96,6 +100,12 @@ func metaPath(stateDir string) string { return filepath.Join(stateDir, "meta.jso | |||||
| 96 | // write is atomic (tmp + rename) so a crash mid-write can't leave a | 100 | // write is atomic (tmp + rename) so a crash mid-write can't leave a | ||
| 97 | // half-parsable file that fails Reload. | 101 | // half-parsable file that fails Reload. | ||
| 98 | func writeAgentMeta(stateDir string, a *Agent) error { | 102 | func writeAgentMeta(stateDir string, a *Agent) error { | ||
| 103 | + | if a == nil { | |||
| 104 | + | return errors.New("subagent meta: nil agent") | |||
| 105 | + | } | |||
| 106 | + | a.persistMu.Lock() | |||
| 107 | + | defer a.persistMu.Unlock() | |||
| 108 | + | ||||
| 99 | fastMode := a.FastMode | 109 | fastMode := a.FastMode | ||
| 100 | webSearchPolicy := childWebSearchPolicy(a.WebSearchPolicy, a.Subagent, a.Tools) | 110 | webSearchPolicy := childWebSearchPolicy(a.WebSearchPolicy, a.Subagent, a.Tools) | ||
| 101 | a.mu.Lock() | 111 | a.mu.Lock() | ||
| @@ -104,6 +114,8 @@ func writeAgentMeta(stateDir string, a *Agent) error { | |||||
| 104 | finished := a.finished | 114 | finished := a.finished | ||
| 105 | resumePrompt := a.resumePromptText | 115 | resumePrompt := a.resumePromptText | ||
| 106 | resumePromptAt := a.resumePromptAt | 116 | resumePromptAt := a.resumePromptAt | ||
| 117 | + | resumePromptID := a.resumePromptID | |||
| 118 | + | resumeQueue := append([]queuedResumePrompt(nil), a.resumePromptQueue...) | |||
| 107 | errText := "" | 119 | errText := "" | ||
| 108 | if a.lastErr != nil { | 120 | if a.lastErr != nil { | ||
| 109 | errText = a.lastErr.Error() | 121 | errText = a.lastErr.Error() | ||
| @@ -113,6 +125,8 @@ func writeAgentMeta(stateDir string, a *Agent) error { | |||||
| 113 | processState := a.processState | 125 | processState := a.processState | ||
| 114 | turnState := a.turnState | 126 | turnState := a.turnState | ||
| 115 | currentTurnID := a.currentTurnID | 127 | currentTurnID := a.currentTurnID | ||
| 128 | + | lifetimeTurns := a.LifetimeTurns | |||
| 129 | + | currentRunTurns := a.CurrentRunTurns | |||
| 116 | attempt := a.Attempt | 130 | attempt := a.Attempt | ||
| 117 | processPID := a.ProcessPID | 131 | processPID := a.ProcessPID | ||
| 118 | updatedAt := a.updatedAt | 132 | updatedAt := a.updatedAt | ||
| @@ -143,6 +157,8 @@ func writeAgentMeta(stateDir string, a *Agent) error { | |||||
| 143 | WorkspaceBase: a.WorkspaceBase, | 157 | WorkspaceBase: a.WorkspaceBase, | ||
| 144 | WorkspaceCapture: a.WorkspaceCapture, | 158 | WorkspaceCapture: a.WorkspaceCapture, | ||
| 145 | MaxTurns: a.MaxTurns, | 159 | MaxTurns: a.MaxTurns, | ||
| 160 | + | LifetimeTurns: lifetimeTurns, | |||
| 161 | + | CurrentRunTurns: currentRunTurns, | |||
| 146 | Timeout: a.Timeout, | 162 | Timeout: a.Timeout, | ||
| 147 | Tools: append([]string(nil), a.Tools...), | 163 | Tools: append([]string(nil), a.Tools...), | ||
| 148 | WebSearchPolicy: webSearchPolicy, | 164 | WebSearchPolicy: webSearchPolicy, | ||
| @@ -165,6 +181,8 @@ func writeAgentMeta(stateDir string, a *Agent) error { | |||||
| 165 | SessionPath: a.SessionPath, | 181 | SessionPath: a.SessionPath, | ||
| 166 | ResumePrompt: resumePrompt, | 182 | ResumePrompt: resumePrompt, | ||
| 167 | ResumePromptAt: resumePromptAt, | 183 | ResumePromptAt: resumePromptAt, | ||
| 184 | + | ResumePromptID: resumePromptID, | |||
| 185 | + | ResumeQueue: resumeQueue, | |||
| 168 | SessionID: a.SessionID, | 186 | SessionID: a.SessionID, | ||
| 169 | } | 187 | } | ||
| 170 | b, err := json.MarshalIndent(m, "", " ") | 188 | b, err := json.MarshalIndent(m, "", " ") | ||
| @@ -175,14 +193,50 @@ func writeAgentMeta(stateDir string, a *Agent) error { | |||||
| 175 | return fmt.Errorf("subagents meta dir: %w", err) | 193 | return fmt.Errorf("subagents meta dir: %w", err) | ||
| 176 | } | 194 | } | ||
| 177 | final := metaPath(stateDir) | 195 | final := metaPath(stateDir) | ||
| 178 | - | tmp := final + ".tmp" | 196 | + | // Use a unique temporary file while holding the per-agent persistence |
| 179 | - | if err := os.WriteFile(tmp, append(b, '\n'), 0o600); err != nil { | 197 | + | // lock. A fixed meta.json.tmp is vulnerable to a concurrent writer (or a |
| 198 | + | // stale file after a crash) replacing another writer's snapshot. | |||
| 199 | + | tmpFile, err := os.CreateTemp(stateDir, ".meta.json.tmp-*") | |||
| 200 | + | if err != nil { | |||
| 201 | + | return fmt.Errorf("subagents meta temp: %w", err) | |||
| 202 | + | } | |||
| 203 | + | tmp := tmpFile.Name() | |||
| 204 | + | removeTemp := true | |||
| 205 | + | defer func() { | |||
| 206 | + | _ = tmpFile.Close() | |||
| 207 | + | if removeTemp { | |||
| 208 | + | _ = os.Remove(tmp) | |||
| 209 | + | } | |||
| 210 | + | }() | |||
| 211 | + | if err := tmpFile.Chmod(0o600); err != nil { | |||
| 212 | + | return fmt.Errorf("subagents meta permissions: %w", err) | |||
| 213 | + | } | |||
| 214 | + | if _, err := tmpFile.Write(append(b, '\n')); err != nil { | |||
| 180 | return fmt.Errorf("subagents meta write: %w", err) | 215 | return fmt.Errorf("subagents meta write: %w", err) | ||
| 181 | } | 216 | } | ||
| 217 | + | if err := tmpFile.Sync(); err != nil { | |||
| 218 | + | return fmt.Errorf("subagents meta sync: %w", err) | |||
| 219 | + | } | |||
| 220 | + | if err := tmpFile.Close(); err != nil { | |||
| 221 | + | return fmt.Errorf("subagents meta close: %w", err) | |||
| 222 | + | } | |||
| 182 | if err := os.Rename(tmp, final); err != nil { | 223 | if err := os.Rename(tmp, final); err != nil { | ||
| 183 | - | _ = os.Remove(tmp) | |||
| 184 | return fmt.Errorf("subagent meta rename: %w", err) | 224 | return fmt.Errorf("subagent meta rename: %w", err) | ||
| 185 | } | 225 | } | ||
| 226 | + | removeTemp = false | |||
| 227 | + | // Sync the containing directory so the rename itself survives a host | |||
| 228 | + | // crash, not just the bytes in the temporary file. | |||
| 229 | + | dir, err := os.Open(stateDir) | |||
| 230 | + | if err != nil { | |||
| 231 | + | return fmt.Errorf("subagents meta directory open: %w", err) | |||
| 232 | + | } | |||
| 233 | + | if err := dir.Sync(); err != nil { | |||
| 234 | + | _ = dir.Close() | |||
| 235 | + | return fmt.Errorf("subagents meta directory sync: %w", err) | |||
| 236 | + | } | |||
| 237 | + | if err := dir.Close(); err != nil { | |||
| 238 | + | return fmt.Errorf("subagents meta directory close: %w", err) | |||
| 239 | + | } | |||
| 186 | return nil | 240 | return nil | ||
| 187 | } | 241 | } | ||
| 188 | 242 | ||||
| @@ -420,57 +474,61 @@ func (f *Supervisor) buildDetachedAgent(m agentMeta) *Agent { | |||||
| 420 | agentStateDir = filepath.Dir(m.EventLogPath) | 474 | agentStateDir = filepath.Dir(m.EventLogPath) | ||
| 421 | } | 475 | } | ||
| 422 | a := &Agent{ | 476 | a := &Agent{ | ||
| 423 | - | ID: m.ID, | 477 | + | ID: m.ID, |
| 424 | - | Task: m.Task, | 478 | + | Task: m.Task, |
| 425 | - | OriginalTask: firstNonEmpty(m.OriginalTask, m.Task), | 479 | + | OriginalTask: firstNonEmpty(m.OriginalTask, m.Task), |
| 426 | - | Dir: dir, | 480 | + | Dir: dir, |
| 427 | - | RepositoryRoot: firstNonEmpty(m.RepositoryRoot, f.cfg.RepoRoot), | 481 | + | RepositoryRoot: firstNonEmpty(m.RepositoryRoot, f.cfg.RepoRoot), |
| 428 | - | Started: m.Started, | 482 | + | Started: m.Started, |
| 429 | - | ParentID: m.ParentID, | 483 | + | ParentID: m.ParentID, |
| 430 | - | BatchID: m.BatchID, | 484 | + | BatchID: m.BatchID, |
| 431 | - | RootSessionID: m.RootSessionID, | 485 | + | RootSessionID: m.RootSessionID, |
| 432 | - | Model: m.Model, | 486 | + | Model: m.Model, |
| 433 | - | Provider: m.Provider, | 487 | + | Provider: m.Provider, |
| 434 | - | BaseURL: m.BaseURL, | 488 | + | BaseURL: m.BaseURL, |
| 435 | - | InsecureTLS: m.InsecureTLS, | 489 | + | InsecureTLS: m.InsecureTLS, |
| 436 | - | Reasoning: m.Reasoning, | 490 | + | Reasoning: m.Reasoning, |
| 437 | - | FastMode: fastMode, | 491 | + | FastMode: fastMode, |
| 438 | - | Subagent: m.Subagent, | 492 | + | Subagent: m.Subagent, |
| 439 | - | WorkspaceMode: m.WorkspaceMode, | 493 | + | WorkspaceMode: m.WorkspaceMode, |
| 440 | - | WorkspacePath: m.WorkspacePath, | 494 | + | WorkspacePath: m.WorkspacePath, |
| 441 | - | WorkspaceBase: m.WorkspaceBase, | 495 | + | WorkspaceBase: m.WorkspaceBase, |
| 442 | - | WorkspaceCapture: m.WorkspaceCapture, | 496 | + | WorkspaceCapture: m.WorkspaceCapture, |
| 443 | - | MaxTurns: m.MaxTurns, | 497 | + | MaxTurns: m.MaxTurns, |
| 444 | - | Timeout: m.Timeout, | 498 | + | LifetimeTurns: m.LifetimeTurns, |
| 445 | - | Tools: append([]string(nil), m.Tools...), | 499 | + | CurrentRunTurns: m.CurrentRunTurns, |
| 446 | - | WebSearchPolicy: childWebSearchPolicy(m.WebSearchPolicy, m.Subagent, m.Tools), | 500 | + | Timeout: m.Timeout, |
| 447 | - | Attempt: m.Attempt, | 501 | + | Tools: append([]string(nil), m.Tools...), |
| 448 | - | ProcessPID: m.ProcessPID, | 502 | + | WebSearchPolicy: childWebSearchPolicy(m.WebSearchPolicy, m.Subagent, m.Tools), |
| 449 | - | InboxPath: m.InboxPath, | 503 | + | Attempt: m.Attempt, |
| 450 | - | EventLogPath: m.EventLogPath, | 504 | + | ProcessPID: m.ProcessPID, |
| 451 | - | SessionPath: m.SessionPath, | 505 | + | InboxPath: m.InboxPath, |
| 452 | - | resumePromptText: m.ResumePrompt, | 506 | + | EventLogPath: m.EventLogPath, |
| 453 | - | resumePromptAt: m.ResumePromptAt, | 507 | + | SessionPath: m.SessionPath, |
| 454 | - | SessionID: m.SessionID, | 508 | + | resumePromptText: m.ResumePrompt, |
| 455 | - | inbox: NewInbox(m.InboxPath), | 509 | + | resumePromptAt: m.ResumePromptAt, |
| 456 | - | status: status, | 510 | + | resumePromptID: m.ResumePromptID, |
| 457 | - | activity: activity, | 511 | + | resumePromptQueue: append([]queuedResumePrompt(nil), m.ResumeQueue...), |
| 458 | - | finished: m.Finished, | 512 | + | SessionID: m.SessionID, |
| 459 | - | lastErr: lastErr, | 513 | + | inbox: NewInbox(m.InboxPath), |
| 460 | - | legacyEventState: legacyEventState, | 514 | + | status: status, |
| 461 | - | processState: processState, | 515 | + | activity: activity, |
| 462 | - | turnState: turnState, | 516 | + | finished: m.Finished, |
| 463 | - | currentTurnID: m.CurrentTurnID, | 517 | + | lastErr: lastErr, |
| 464 | - | updatedAt: m.UpdatedAt, | 518 | + | legacyEventState: legacyEventState, |
| 465 | - | lastActivity: lastActivity, | 519 | + | processState: processState, |
| 466 | - | stateDir: agentStateDir, | 520 | + | turnState: turnState, |
| 467 | - | resultRef: m.ResultRef, | 521 | + | currentTurnID: m.CurrentTurnID, |
| 468 | - | patchRef: m.PatchRef, | 522 | + | updatedAt: m.UpdatedAt, |
| 469 | - | changedFiles: append([]string(nil), m.ChangedFiles...), | 523 | + | lastActivity: lastActivity, |
| 470 | - | maxOutputBytes: f.cfg.Policy.MaxOutputBytes, | 524 | + | stateDir: agentStateDir, |
| 471 | - | maxOutputLines: f.cfg.Policy.MaxOutputLines, | 525 | + | resultRef: m.ResultRef, |
| 472 | - | done: make(chan struct{}), | 526 | + | patchRef: m.PatchRef, |
| 473 | - | turnResults: make(chan *TurnResult, 16), | 527 | + | changedFiles: append([]string(nil), m.ChangedFiles...), |
| 528 | + | maxOutputBytes: f.cfg.Policy.MaxOutputBytes, | |||
| 529 | + | maxOutputLines: f.cfg.Policy.MaxOutputLines, | |||
| 530 | + | done: make(chan struct{}), | |||
| 531 | + | turnResults: make(chan *TurnResult, 16), | |||
| 474 | } | 532 | } | ||
| 475 | if a.updatedAt.IsZero() { | 533 | if a.updatedAt.IsZero() { | ||
| 476 | a.updatedAt = lastActivity | 534 | a.updatedAt = lastActivity | ||
| @@ -542,7 +600,7 @@ func (a *Agent) loadTranscript() error { | |||||
| 542 | if err != nil { | 600 | if err != nil { | ||
| 543 | return fmt.Errorf("read subagent transcript: %w", err) | 601 | return fmt.Errorf("read subagent transcript: %w", err) | ||
| 544 | } | 602 | } | ||
| 545 | - | if prompt, acceptedAt := a.ResumePromptInfo(); prompt != "" && resumePromptAcknowledged(events, acceptedAt) { | 603 | + | if prompt, acceptedAt := a.ResumePromptInfo(); prompt != "" && resumePromptAcknowledged(events, prompt, acceptedAt) { |
| 546 | a.clearResumePrompt() | 604 | a.clearResumePrompt() | ||
| 547 | } | 605 | } | ||
| 548 | a.mu.Lock() | 606 | a.mu.Lock() | ||
| @@ -583,12 +641,23 @@ func (a *Agent) loadTranscript() error { | |||||
| 583 | // Status precedence: explicit lifecycle events (agent_stopped) win | 641 | // Status precedence: explicit lifecycle events (agent_stopped) win | ||
| 584 | // over inferred ones (assistant_message → idle). If the log contains | 642 | // over inferred ones (assistant_message → idle). If the log contains | ||
| 585 | // no terminator we keep status=StatusDetached so the user can resume. | 643 | // no terminator we keep status=StatusDetached so the user can resume. | ||
| 586 | - | func resumePromptAcknowledged(events []Event, acceptedAt time.Time) bool { | 644 | + | func resumePromptAcknowledged(events []Event, prompt string, acceptedAt time.Time) bool { |
| 587 | - | if acceptedAt.IsZero() { | 645 | + | if prompt == "" || acceptedAt.IsZero() { |
| 588 | return false | 646 | return false | ||
| 589 | } | 647 | } | ||
| 648 | + | userDurable := false | |||
| 590 | for _, ev := range events { | 649 | for _, ev := range events { | ||
| 591 | - | if (ev.Type == EventTurnStarted || ev.Type == "turn_start") && !ev.Time.Before(acceptedAt) { | 650 | + | if ev.Time.Before(acceptedAt) { |
| 651 | + | continue | |||
| 652 | + | } | |||
| 653 | + | if ev.Type == "user_message" && userMessageText(ev) == prompt { | |||
| 654 | + | // The worker emits this event only after the matching session | |||
| 655 | + | // message has been appended and flushed. A crash here leaves the | |||
| 656 | + | // prompt pending, which is intentionally safe to replay. | |||
| 657 | + | userDurable = true | |||
| 658 | + | continue | |||
| 659 | + | } | |||
| 660 | + | if userDurable && isDelegatedTurnStart(ev) { | |||
| 592 | return true | 661 | return true | ||
| 593 | } | 662 | } | ||
| 594 | } | 663 | } | ||
| @@ -599,14 +668,11 @@ func replayEventsIntoAgent(a *Agent, evs []Event) { | |||||
| 599 | terminal := false | 668 | terminal := false | ||
| 600 | for _, ev := range evs { | 669 | for _, ev := range evs { | ||
| 601 | switch ev.Type { | 670 | switch ev.Type { | ||
| 602 | - | case EventTurnStarted, "turn_start": | 671 | + | case EventTurnStarted: |
| 603 | - | turnID := ev.TurnID | 672 | + | if !isDelegatedTurnStart(ev) { |
| 604 | - | if turnID == "" { | 673 | + | continue |
| 605 | - | if step, ok := ev.Data["step"].(float64); ok { | |||
| 606 | - | turnID = fmt.Sprintf("turn-%d", int(step)) | |||
| 607 | - | } | |||
| 608 | } | 674 | } | ||
| 609 | - | a.setTurnState(TurnRunning, turnID) | 675 | + | a.setTurnState(TurnRunning, eventTurnID(ev)) |
| 610 | case EventTurnResult, "turn_result": | 676 | case EventTurnResult, "turn_result": | ||
| 611 | if result, err := decodeTurnResultEvent(ev, a.ID, a.maxOutputBytes, a.maxOutputLines); err == nil { | 677 | if result, err := decodeTurnResultEvent(ev, a.ID, a.maxOutputBytes, a.maxOutputLines); err == nil { | ||
| 612 | a.setResult(result) | 678 | a.setResult(result) | ||
| @@ -735,9 +801,16 @@ func (f *Supervisor) ResumeSession(ctx context.Context, id string) (*Agent, erro | |||||
| 735 | 801 | ||||
| 736 | // ResumeWithPrompt continues an agent with a manager follow-up while retaining | 802 | // ResumeWithPrompt continues an agent with a manager follow-up while retaining | ||
| 737 | // its existing session. An idle live worker receives the turn through its | 803 | // its existing session. An idle live worker receives the turn through its | ||
| 738 | - | // inbox; a terminal worker is restarted with the follow-up as its initial | 804 | + | // inbox; an active live worker queues the follow-up for the next idle turn; a |
| 739 | - | // prompt, avoiding a race with inbox listener startup. | 805 | + | // terminal worker is restarted with the follow-up as its initial prompt, |
| 806 | + | // avoiding a race with inbox listener startup. | |||
| 740 | func (f *Supervisor) ResumeWithPrompt(ctx context.Context, id, prompt string) (*Agent, error) { | 807 | func (f *Supervisor) ResumeWithPrompt(ctx context.Context, id, prompt string) (*Agent, error) { | ||
| 808 | + | if ctx == nil { | |||
| 809 | + | ctx = context.Background() | |||
| 810 | + | } | |||
| 811 | + | if err := ctx.Err(); err != nil { | |||
| 812 | + | return nil, err | |||
| 813 | + | } | |||
| 741 | if strings.TrimSpace(prompt) == "" { | 814 | if strings.TrimSpace(prompt) == "" { | ||
| 742 | return nil, fmt.Errorf("subagent: follow-up prompt is required") | 815 | return nil, fmt.Errorf("subagent: follow-up prompt is required") | ||
| 743 | } | 816 | } | ||
| @@ -750,15 +823,42 @@ func (f *Supervisor) ResumeWithPrompt(ctx context.Context, id, prompt string) (* | |||||
| 750 | } | 823 | } | ||
| 751 | status := existing.Status() | 824 | status := existing.Status() | ||
| 752 | if status == StatusRunning { | 825 | if status == StatusRunning { | ||
| 753 | - | if existing.ProcessState() != ProcessAlive || existing.TurnState() != TurnIdle { | 826 | + | if existing.ProcessState() != ProcessAlive { |
| 754 | f.operationMu.Unlock() | 827 | f.operationMu.Unlock() | ||
| 755 | - | return nil, fmt.Errorf("subagent: agent %s is still working; wait for it to become idle before sending a follow-up", existing.ID) | 828 | + | return nil, fmt.Errorf("subagent: agent %s is still working; wait for its worker to become available before sending a follow-up", existing.ID) |
| 756 | } | 829 | } | ||
| 757 | if existing.inbox == nil { | 830 | if existing.inbox == nil { | ||
| 758 | f.operationMu.Unlock() | 831 | f.operationMu.Unlock() | ||
| 759 | return nil, fmt.Errorf("subagent: agent %s has no inbox", existing.ID) | 832 | return nil, fmt.Errorf("subagent: agent %s has no inbox", existing.ID) | ||
| 760 | } | 833 | } | ||
| 834 | + | if err := ctx.Err(); err != nil { | |||
| 835 | + | f.operationMu.Unlock() | |||
| 836 | + | return nil, err | |||
| 837 | + | } | |||
| 838 | + | if existing.TurnState() != TurnRunning && existing.hasQueuedResumePrompt() { | |||
| 839 | + | f.operationMu.Unlock() | |||
| 840 | + | return nil, fmt.Errorf("subagent: agent %s already has a follow-up pending delivery", existing.ID) | |||
| 841 | + | } | |||
| 842 | + | if existing.TurnState() == TurnRunning { | |||
| 843 | + | // Active turns cannot accept a command through the worker inbox yet. | |||
| 844 | + | // Keep the prompt durable and let the idle transition deliver it in | |||
| 845 | + | // order after the current message turn finishes. | |||
| 846 | + | acceptedAt := f.cfg.Now() | |||
| 847 | + | existing.queueResumePrompt(prompt, acceptedAt) | |||
| 848 | + | if err := f.persistAgent(existing); err != nil { | |||
| 849 | + | existing.removeQueuedResumePrompt(prompt, acceptedAt) | |||
| 850 | + | f.operationMu.Unlock() | |||
| 851 | + | return nil, fmt.Errorf("subagent: persist queued follow-up for %s: %w", existing.ID, err) | |||
| 852 | + | } | |||
| 853 | + | f.operationMu.Unlock() | |||
| 854 | + | return existing, nil | |||
| 855 | + | } | |||
| 856 | + | if existing.TurnState() != TurnIdle { | |||
| 857 | + | f.operationMu.Unlock() | |||
| 858 | + | return nil, fmt.Errorf("subagent: agent %s is still working; wait for it to become idle before sending a follow-up", existing.ID) | |||
| 859 | + | } | |||
| 761 | turnID := existing.CurrentTurnID() | 860 | turnID := existing.CurrentTurnID() | ||
| 861 | + | previousRunTurns := existing.resetCurrentRunTurns() | |||
| 762 | // Reserve the worker before writing to its inbox. A second manager call | 862 | // Reserve the worker before writing to its inbox. A second manager call | ||
| 763 | // must not enqueue another turn in the small interval before the worker | 863 | // must not enqueue another turn in the small interval before the worker | ||
| 764 | // reports its turn.start event. Persist the prompt first so a host exit | 864 | // reports its turn.start event. Persist the prompt first so a host exit | ||
| @@ -766,18 +866,39 @@ func (f *Supervisor) ResumeWithPrompt(ctx context.Context, id, prompt string) (* | |||||
| 766 | previousPrompt, previousPromptAt := existing.setResumePrompt(prompt, f.cfg.Now()) | 866 | previousPrompt, previousPromptAt := existing.setResumePrompt(prompt, f.cfg.Now()) | ||
| 767 | existing.setTurnState(TurnQueued, turnID) | 867 | existing.setTurnState(TurnQueued, turnID) | ||
| 768 | inbox := existing.inbox | 868 | inbox := existing.inbox | ||
| 769 | - | command := NewCommand(CommandTurnStart, existing.ID, turnID, TurnStartPayload{Prompt: prompt}) | 869 | + | command := NewCommand(CommandTurnStart, existing.ID, turnID, TurnStartPayload{Prompt: prompt, NewRun: true}) |
| 770 | - | f.persistAgent(existing) | 870 | + | command.MessageID = existing.resumePromptCommandID() |
| 871 | + | if err := f.persistAgent(existing); err != nil { | |||
| 872 | + | existing.setTurnState(TurnIdle, turnID) | |||
| 873 | + | existing.setCurrentRunTurns(previousRunTurns) | |||
| 874 | + | existing.setResumePrompt(previousPrompt, previousPromptAt) | |||
| 875 | + | f.operationMu.Unlock() | |||
| 876 | + | return nil, fmt.Errorf("subagent: persist follow-up for %s: %w", existing.ID, err) | |||
| 877 | + | } | |||
| 771 | f.operationMu.Unlock() | 878 | f.operationMu.Unlock() | ||
| 772 | 879 | ||||
| 773 | if err := inbox.SendCommandContext(ctx, command); err != nil { | 880 | if err := inbox.SendCommandContext(ctx, command); err != nil { | ||
| 881 | + | retryUnknown := false | |||
| 774 | f.operationMu.Lock() | 882 | f.operationMu.Lock() | ||
| 775 | if existing.Status() == StatusRunning && existing.TurnState() == TurnQueued && existing.CurrentTurnID() == turnID { | 883 | if existing.Status() == StatusRunning && existing.TurnState() == TurnQueued && existing.CurrentTurnID() == turnID { | ||
| 776 | existing.setTurnState(TurnIdle, turnID) | 884 | existing.setTurnState(TurnIdle, turnID) | ||
| 777 | - | existing.setResumePrompt(previousPrompt, previousPromptAt) | 885 | + | existing.setCurrentRunTurns(previousRunTurns) |
| 778 | - | f.persistAgent(existing) | 886 | + | if errors.Is(err, ErrDeliveryUnknown) { |
| 887 | + | existing.rejectActiveResumePrompt() | |||
| 888 | + | retryUnknown = true | |||
| 889 | + | } else { | |||
| 890 | + | existing.setResumePrompt(previousPrompt, previousPromptAt) | |||
| 891 | + | } | |||
| 892 | + | if persistErr := f.persistAgent(existing); persistErr != nil { | |||
| 893 | + | err = fmt.Errorf("%w; persist recovery: %v", err, persistErr) | |||
| 894 | + | } | |||
| 779 | } | 895 | } | ||
| 780 | f.operationMu.Unlock() | 896 | f.operationMu.Unlock() | ||
| 897 | + | if retryUnknown { | |||
| 898 | + | // The command keeps its durable identity. A retry is safe whether | |||
| 899 | + | // the worker parsed the first write or only received a prefix. | |||
| 900 | + | f.dispatchQueuedResume(existing) | |||
| 901 | + | } | |||
| 781 | return nil, fmt.Errorf("subagent: send follow-up to %s: %w", existing.ID, err) | 902 | return nil, fmt.Errorf("subagent: send follow-up to %s: %w", existing.ID, err) | ||
| 782 | } | 903 | } | ||
| 783 | return existing, nil | 904 | return existing, nil | ||
| @@ -787,6 +908,57 @@ func (f *Supervisor) ResumeWithPrompt(ctx context.Context, id, prompt string) (* | |||||
| 787 | return f.resume(ctx, id, true, prompt) | 908 | return f.resume(ctx, id, true, prompt) | ||
| 788 | } | 909 | } | ||
| 789 | 910 | ||||
| 911 | + | // dispatchQueuedResume delivers the oldest follow-up accepted during an | |||
| 912 | + | // active turn after the worker reports that its inbox can accept a new turn. | |||
| 913 | + | // The operation lock serializes this transition with manager resume calls and | |||
| 914 | + | // the agent's durable prompt state prevents a host exit from losing it. | |||
| 915 | + | func (f *Supervisor) dispatchQueuedResume(a *Agent) { | |||
| 916 | + | if a == nil { | |||
| 917 | + | return | |||
| 918 | + | } | |||
| 919 | + | f.operationMu.Lock() | |||
| 920 | + | if a.Status() != StatusRunning || a.ProcessState() != ProcessAlive || a.TurnState() != TurnIdle || a.inbox == nil { | |||
| 921 | + | f.operationMu.Unlock() | |||
| 922 | + | return | |||
| 923 | + | } | |||
| 924 | + | prompt, acceptedAt, commandID, ok := a.startQueuedResumePrompt() | |||
| 925 | + | if !ok { | |||
| 926 | + | f.operationMu.Unlock() | |||
| 927 | + | return | |||
| 928 | + | } | |||
| 929 | + | turnID := a.CurrentTurnID() | |||
| 930 | + | previousRunTurns := a.resetCurrentRunTurns() | |||
| 931 | + | a.setTurnState(TurnQueued, turnID) | |||
| 932 | + | inbox := a.inbox | |||
| 933 | + | command := NewCommand(CommandTurnStart, a.ID, turnID, TurnStartPayload{Prompt: prompt, NewRun: true}) | |||
| 934 | + | command.MessageID = commandID | |||
| 935 | + | if err := f.persistAgent(a); err != nil { | |||
| 936 | + | a.restoreResumePromptToQueue(prompt, acceptedAt) | |||
| 937 | + | a.setCurrentRunTurns(previousRunTurns) | |||
| 938 | + | a.setTurnState(TurnIdle, turnID) | |||
| 939 | + | f.operationMu.Unlock() | |||
| 940 | + | return | |||
| 941 | + | } | |||
| 942 | + | f.operationMu.Unlock() | |||
| 943 | + | ||||
| 944 | + | if err := inbox.SendCommand(command); err == nil { | |||
| 945 | + | return | |||
| 946 | + | } | |||
| 947 | + | ||||
| 948 | + | // A failed delivery must put the prompt back at the head of the queue. | |||
| 949 | + | // Keep the worker idle so a later retry, or an explicit resume, can send | |||
| 950 | + | // it without skipping any subsequent follow-ups. | |||
| 951 | + | f.operationMu.Lock() | |||
| 952 | + | if a.Status() == StatusRunning && a.ProcessState() == ProcessAlive && a.TurnState() == TurnQueued && a.CurrentTurnID() == turnID { | |||
| 953 | + | if a.restoreResumePromptToQueue(prompt, acceptedAt) { | |||
| 954 | + | a.setCurrentRunTurns(previousRunTurns) | |||
| 955 | + | a.setTurnState(TurnIdle, turnID) | |||
| 956 | + | f.persistAgent(a) | |||
| 957 | + | } | |||
| 958 | + | } | |||
| 959 | + | f.operationMu.Unlock() | |||
| 960 | + | } | |||
| 961 | + | ||||
| 790 | // RestartTask intentionally runs the stored original task again in a fresh | 962 | // RestartTask intentionally runs the stored original task again in a fresh | ||
| 791 | // worker attempt. Callers should use ResumeSession for normal recovery. | 963 | // worker attempt. Callers should use ResumeSession for normal recovery. | ||
| 792 | func (f *Supervisor) RestartTask(ctx context.Context, id string) (*Agent, error) { | 964 | func (f *Supervisor) RestartTask(ctx context.Context, id string) (*Agent, error) { | ||
| @@ -852,10 +1024,35 @@ func (f *Supervisor) resume(ctx context.Context, id string, resuming bool, resum | |||||
| 852 | } | 1024 | } | ||
| 853 | now := f.cfg.Now() | 1025 | now := f.cfg.Now() | ||
| 854 | fastMode := existing.FastMode | 1026 | fastMode := existing.FastMode | ||
| 1027 | + | lifetimeTurns := existing.LifetimeTurnsValue() | |||
| 1028 | + | currentRunTurns := existing.CurrentRunTurnsValue() | |||
| 855 | pendingResumePrompt, pendingResumePromptAt := existing.ResumePromptInfo() | 1029 | pendingResumePrompt, pendingResumePromptAt := existing.ResumePromptInfo() | ||
| 1030 | + | pendingResumeQueue := existing.resumePromptQueueSnapshot() | |||
| 856 | if resumePrompt != "" { | 1031 | if resumePrompt != "" { | ||
| 857 | - | pendingResumePrompt = resumePrompt | 1032 | + | currentRunTurns = 0 |
| 858 | - | pendingResumePromptAt = now | 1033 | + | if pendingResumePrompt == "" { |
| 1034 | + | pendingResumePrompt = resumePrompt | |||
| 1035 | + | pendingResumePromptAt = now | |||
| 1036 | + | } else { | |||
| 1037 | + | pendingResumeQueue = append(pendingResumeQueue, queuedResumePrompt{Prompt: resumePrompt, AcceptedAt: now}) | |||
| 1038 | + | } | |||
| 1039 | + | } | |||
| 1040 | + | if resuming && resumePrompt == "" && pendingResumePrompt == "" && len(pendingResumeQueue) > 0 { | |||
| 1041 | + | // A host can exit after accepting an active-worker follow-up but before | |||
| 1042 | + | // dispatching it. Make the first durable queued prompt the resumed | |||
| 1043 | + | // worker's initial turn rather than leaving the queue stranded. | |||
| 1044 | + | pendingResumePrompt = pendingResumeQueue[0].Prompt | |||
| 1045 | + | pendingResumePromptAt = pendingResumeQueue[0].AcceptedAt | |||
| 1046 | + | pendingResumeQueue = pendingResumeQueue[1:] | |||
| 1047 | + | } | |||
| 1048 | + | if resuming && pendingResumePrompt != "" { | |||
| 1049 | + | // A retained follow-up is an explicit new run, including a queued | |||
| 1050 | + | // follow-up recovered after a host restart. Preserve lifetime turns but | |||
| 1051 | + | // never carry the previous run's allowance into this one. | |||
| 1052 | + | currentRunTurns = 0 | |||
| 1053 | + | } | |||
| 1054 | + | if !resuming { | |||
| 1055 | + | currentRunTurns = 0 | |||
| 859 | } | 1056 | } | ||
| 860 | m := agentMeta{ | 1057 | m := agentMeta{ | ||
| 861 | ID: existing.ID, Task: existing.Task, | 1058 | ID: existing.ID, Task: existing.Task, | ||
| @@ -868,7 +1065,7 @@ func (f *Supervisor) resume(ctx context.Context, id string, resuming bool, resum | |||||
| 868 | Subagent: existing.Subagent, | 1065 | Subagent: existing.Subagent, | ||
| 869 | WorkspaceMode: existing.WorkspaceMode, WorkspacePath: existing.WorkspacePath, | 1066 | WorkspaceMode: existing.WorkspaceMode, WorkspacePath: existing.WorkspacePath, | ||
| 870 | WorkspaceBase: existing.WorkspaceBase, WorkspaceCapture: existing.WorkspaceCapture, | 1067 | WorkspaceBase: existing.WorkspaceBase, WorkspaceCapture: existing.WorkspaceCapture, | ||
| 871 | - | MaxTurns: existing.MaxTurns, Timeout: existing.Timeout, Tools: append([]string(nil), existing.Tools...), | 1068 | + | MaxTurns: existing.MaxTurns, LifetimeTurns: lifetimeTurns, CurrentRunTurns: currentRunTurns, Timeout: existing.Timeout, Tools: append([]string(nil), existing.Tools...), |
| 872 | WebSearchPolicy: childWebSearchPolicy(existing.WebSearchPolicy, existing.Subagent, existing.Tools), | 1069 | WebSearchPolicy: childWebSearchPolicy(existing.WebSearchPolicy, existing.Subagent, existing.Tools), | ||
| 873 | CurrentTurnID: existingSnapshot.CurrentTurnID, Attempt: existingSnapshot.Attempt, | 1070 | CurrentTurnID: existingSnapshot.CurrentTurnID, Attempt: existingSnapshot.Attempt, | ||
| 874 | SessionID: existing.SessionID, | 1071 | SessionID: existing.SessionID, | ||
| @@ -876,6 +1073,8 @@ func (f *Supervisor) resume(ctx context.Context, id string, resuming bool, resum | |||||
| 876 | SessionPath: existing.SessionPath, | 1073 | SessionPath: existing.SessionPath, | ||
| 877 | ResumePrompt: pendingResumePrompt, | 1074 | ResumePrompt: pendingResumePrompt, | ||
| 878 | ResumePromptAt: pendingResumePromptAt, | 1075 | ResumePromptAt: pendingResumePromptAt, | ||
| 1076 | + | ResumePromptID: existing.resumePromptCommandID(), | |||
| 1077 | + | ResumeQueue: pendingResumeQueue, | |||
| 879 | } | 1078 | } | ||
| 880 | 1079 | ||||
| 881 | if len(m.Tools) == 0 && len(f.cfg.Policy.AllowedTools) > 0 { | 1080 | if len(m.Tools) == 0 && len(f.cfg.Policy.AllowedTools) > 0 { | ||
| @@ -954,6 +1153,8 @@ func (f *Supervisor) resume(ctx context.Context, id string, resuming bool, resum | |||||
| 954 | WorkspaceBase: m.WorkspaceBase, | 1153 | WorkspaceBase: m.WorkspaceBase, | ||
| 955 | WorkspaceCapture: m.WorkspaceCapture, | 1154 | WorkspaceCapture: m.WorkspaceCapture, | ||
| 956 | MaxTurns: maxTurns, | 1155 | MaxTurns: maxTurns, | ||
| 1156 | + | LifetimeTurns: m.LifetimeTurns, | |||
| 1157 | + | CurrentRunTurns: m.CurrentRunTurns, | |||
| 957 | Timeout: timeout, | 1158 | Timeout: timeout, | ||
| 958 | Attempt: m.Attempt, | 1159 | Attempt: m.Attempt, | ||
| 959 | HeartbeatInterval: f.cfg.Policy.HeartbeatInterval, | 1160 | HeartbeatInterval: f.cfg.Policy.HeartbeatInterval, | ||
| @@ -965,6 +1166,8 @@ func (f *Supervisor) resume(ctx context.Context, id string, resuming bool, resum | |||||
| 965 | Resuming: resuming, | 1166 | Resuming: resuming, | ||
| 966 | resumePromptText: pendingResumePrompt, | 1167 | resumePromptText: pendingResumePrompt, | ||
| 967 | resumePromptAt: pendingResumePromptAt, | 1168 | resumePromptAt: pendingResumePromptAt, | ||
| 1169 | + | resumePromptID: m.ResumePromptID, | |||
| 1170 | + | resumePromptQueue: append([]queuedResumePrompt(nil), pendingResumeQueue...), | |||
| 968 | inbox: NewInbox(m.InboxPath), | 1171 | inbox: NewInbox(m.InboxPath), | ||
| 969 | status: StatusPending, | 1172 | status: StatusPending, | ||
| 970 | activity: "resuming", | 1173 | activity: "resuming", | ||
| @@ -989,6 +1192,7 @@ func (f *Supervisor) resume(ctx context.Context, id string, resuming bool, resum | |||||
| 989 | } | 1192 | } | ||
| 990 | a.ctx, a.cancel = runCtx, cancel | 1193 | a.ctx, a.cancel = runCtx, cancel | ||
| 991 | a.persistFn = f.persistAgent | 1194 | a.persistFn = f.persistAgent | ||
| 1195 | + | a.setOnTurnIdle(func() { f.dispatchQueuedResume(a) }) | |||
| 992 | a.workspaceCleanup = func() error { return workspace.Cleanup(context.Background()) } | 1196 | a.workspaceCleanup = func() error { return workspace.Cleanup(context.Background()) } | ||
| 993 | a.workspaceCapture = func() (WorkspaceCapture, error) { return workspace.Capture(context.Background()) } | 1197 | a.workspaceCapture = func() (WorkspaceCapture, error) { return workspace.Capture(context.Background()) } | ||
| 994 | a.runner = f.cfg.NewRunner(a) | 1198 | a.runner = f.cfg.NewRunner(a) | ||
M packages/agent/subagents/persist_test.go +32 -1
| @@ -14,6 +14,34 @@ import ( | |||
| 14 | 14 | "time" | |
| 15 | 15 | ) | |
| 16 | 16 | ||
| 17 | + | func TestWriteAgentMetaSerializesConcurrentSnapshots(t *testing.T) { | |
| 18 | + | stateDir := t.TempDir() | |
| 19 | + | a := &Agent{ID: "concurrent-meta", Task: "retain queued prompts", stateDir: stateDir} | |
| 20 | + | var wg sync.WaitGroup | |
| 21 | + | for i := 0; i < 32; i++ { | |
| 22 | + | wg.Add(1) | |
| 23 | + | go func(i int) { | |
| 24 | + | defer wg.Done() | |
| 25 | + | a.setTurnCounts(i, i/2) | |
| 26 | + | a.queueResumePrompt("prompt", time.Now()) | |
| 27 | + | if err := writeAgentMeta(stateDir, a); err != nil { | |
| 28 | + | t.Errorf("write metadata: %v", err) | |
| 29 | + | } | |
| 30 | + | }(i) | |
| 31 | + | } | |
| 32 | + | wg.Wait() | |
| 33 | + | if _, err := readAgentMeta(stateDir); err != nil { | |
| 34 | + | t.Fatalf("final metadata is not a complete JSON snapshot: %v", err) | |
| 35 | + | } | |
| 36 | + | matches, err := filepath.Glob(filepath.Join(stateDir, ".meta.json.tmp-*")) | |
| 37 | + | if err != nil { | |
| 38 | + | t.Fatal(err) | |
| 39 | + | } | |
| 40 | + | if len(matches) != 0 { | |
| 41 | + | t.Fatalf("temporary metadata files remain: %v", matches) | |
| 42 | + | } | |
| 43 | + | } | |
| 44 | + | ||
| 17 | 45 | // TestSpawnWritesMetaJSON asserts the durability contract: every | |
| 18 | 46 | // successful Spawn leaves a meta.json on disk with the agent's | |
| 19 | 47 | // identity bits. Without this, Reload on the next zut launch can't | |
| @@ -662,7 +690,7 @@ func TestReloadClearsResumePromptAcknowledgedInEventLog(t *testing.T) { | |||
| 662 | 690 | EventLogPath: filepath.Join(stateDir, "events.jsonl"), | |
| 663 | 691 | SessionPath: filepath.Join(stateDir, "session.json"), | |
| 664 | 692 | ResumePrompt: "I applied your review. What do you think now?", | |
| 665 | - | ResumePromptAt: time.Now(), | |
| 693 | + | ResumePromptAt: time.Now().Add(-time.Second), | |
| 666 | 694 | } | |
| 667 | 695 | data, err := json.Marshal(m) | |
| 668 | 696 | if err != nil { | |
| @@ -675,6 +703,9 @@ func TestReloadClearsResumePromptAcknowledgedInEventLog(t *testing.T) { | |||
| 675 | 703 | if err != nil { | |
| 676 | 704 | t.Fatal(err) | |
| 677 | 705 | } | |
| 706 | + | if err := log.Append(NewEvent("user_message", map[string]any{"content": []any{map[string]any{"type": "text", "text": m.ResumePrompt}}})); err != nil { | |
| 707 | + | t.Fatal(err) | |
| 708 | + | } | |
| 678 | 709 | if err := log.Append(NewEvent(EventTurnStarted, map[string]any{"step": float64(1)})); err != nil { | |
| 679 | 710 | t.Fatal(err) | |
| 680 | 711 | } | |
| @@ -14,6 +14,34 @@ import ( | |||||
| 14 | "time" | 14 | "time" | ||
| 15 | ) | 15 | ) | ||
| 16 | 16 | ||||
| 17 | + | func TestWriteAgentMetaSerializesConcurrentSnapshots(t *testing.T) { | |||
| 18 | + | stateDir := t.TempDir() | |||
| 19 | + | a := &Agent{ID: "concurrent-meta", Task: "retain queued prompts", stateDir: stateDir} | |||
| 20 | + | var wg sync.WaitGroup | |||
| 21 | + | for i := 0; i < 32; i++ { | |||
| 22 | + | wg.Add(1) | |||
| 23 | + | go func(i int) { | |||
| 24 | + | defer wg.Done() | |||
| 25 | + | a.setTurnCounts(i, i/2) | |||
| 26 | + | a.queueResumePrompt("prompt", time.Now()) | |||
| 27 | + | if err := writeAgentMeta(stateDir, a); err != nil { | |||
| 28 | + | t.Errorf("write metadata: %v", err) | |||
| 29 | + | } | |||
| 30 | + | }(i) | |||
| 31 | + | } | |||
| 32 | + | wg.Wait() | |||
| 33 | + | if _, err := readAgentMeta(stateDir); err != nil { | |||
| 34 | + | t.Fatalf("final metadata is not a complete JSON snapshot: %v", err) | |||
| 35 | + | } | |||
| 36 | + | matches, err := filepath.Glob(filepath.Join(stateDir, ".meta.json.tmp-*")) | |||
| 37 | + | if err != nil { | |||
| 38 | + | t.Fatal(err) | |||
| 39 | + | } | |||
| 40 | + | if len(matches) != 0 { | |||
| 41 | + | t.Fatalf("temporary metadata files remain: %v", matches) | |||
| 42 | + | } | |||
| 43 | + | } | |||
| 44 | + | ||||
| 17 | // TestSpawnWritesMetaJSON asserts the durability contract: every | 45 | // TestSpawnWritesMetaJSON asserts the durability contract: every | ||
| 18 | // successful Spawn leaves a meta.json on disk with the agent's | 46 | // successful Spawn leaves a meta.json on disk with the agent's | ||
| 19 | // identity bits. Without this, Reload on the next zut launch can't | 47 | // identity bits. Without this, Reload on the next zut launch can't | ||
| @@ -662,7 +690,7 @@ func TestReloadClearsResumePromptAcknowledgedInEventLog(t *testing.T) { | |||||
| 662 | EventLogPath: filepath.Join(stateDir, "events.jsonl"), | 690 | EventLogPath: filepath.Join(stateDir, "events.jsonl"), | ||
| 663 | SessionPath: filepath.Join(stateDir, "session.json"), | 691 | SessionPath: filepath.Join(stateDir, "session.json"), | ||
| 664 | ResumePrompt: "I applied your review. What do you think now?", | 692 | ResumePrompt: "I applied your review. What do you think now?", | ||
| 665 | - | ResumePromptAt: time.Now(), | 693 | + | ResumePromptAt: time.Now().Add(-time.Second), |
| 666 | } | 694 | } | ||
| 667 | data, err := json.Marshal(m) | 695 | data, err := json.Marshal(m) | ||
| 668 | if err != nil { | 696 | if err != nil { | ||
| @@ -675,6 +703,9 @@ func TestReloadClearsResumePromptAcknowledgedInEventLog(t *testing.T) { | |||||
| 675 | if err != nil { | 703 | if err != nil { | ||
| 676 | t.Fatal(err) | 704 | t.Fatal(err) | ||
| 677 | } | 705 | } | ||
| 706 | + | if err := log.Append(NewEvent("user_message", map[string]any{"content": []any{map[string]any{"type": "text", "text": m.ResumePrompt}}})); err != nil { | |||
| 707 | + | t.Fatal(err) | |||
| 708 | + | } | |||
| 678 | if err := log.Append(NewEvent(EventTurnStarted, map[string]any{"step": float64(1)})); err != nil { | 709 | if err := log.Append(NewEvent(EventTurnStarted, map[string]any{"step": float64(1)})); err != nil { | ||
| 679 | t.Fatal(err) | 710 | t.Fatal(err) | ||
| 680 | } | 711 | } | ||
M packages/agent/subagents/protocol.go +4 -0
| @@ -579,6 +579,10 @@ type AgentHeartbeatPayload struct { | |||
| 579 | 579 | } | |
| 580 | 580 | type TurnStartPayload struct { | |
| 581 | 581 | Prompt string `json:"prompt"` | |
| 582 | + | // NewRun starts a fresh max_turns budget while retaining the worker's | |
| 583 | + | // existing session and transcript. It is set only for explicit | |
| 584 | + | // subagent_resume follow-ups. | |
| 585 | + | NewRun bool `json:"new_run,omitempty"` | |
| 582 | 586 | } | |
| 583 | 587 | type TurnCancelPayload struct { | |
| 584 | 588 | Reason string `json:"reason,omitempty"` | |
| @@ -579,6 +579,10 @@ type AgentHeartbeatPayload struct { | |||||
| 579 | } | 579 | } | ||
| 580 | type TurnStartPayload struct { | 580 | type TurnStartPayload struct { | ||
| 581 | Prompt string `json:"prompt"` | 581 | Prompt string `json:"prompt"` | ||
| 582 | + | // NewRun starts a fresh max_turns budget while retaining the worker's | |||
| 583 | + | // existing session and transcript. It is set only for explicit | |||
| 584 | + | // subagent_resume follow-ups. | |||
| 585 | + | NewRun bool `json:"new_run,omitempty"` | |||
| 582 | } | 586 | } | ||
| 583 | type TurnCancelPayload struct { | 587 | type TurnCancelPayload struct { | ||
| 584 | Reason string `json:"reason,omitempty"` | 588 | Reason string `json:"reason,omitempty"` | ||
M packages/agent/subagents/resume_prompt_test.go +137 -0
| @@ -2,12 +2,139 @@ package subagents | |||
| 2 | 2 | ||
| 3 | 3 | import ( | |
| 4 | 4 | "context" | |
| 5 | + | "errors" | |
| 5 | 6 | "path/filepath" | |
| 6 | 7 | "runtime" | |
| 7 | 8 | "testing" | |
| 8 | 9 | "time" | |
| 9 | 10 | ) | |
| 10 | 11 | ||
| 12 | + | func TestResumeWithPromptQueuesFollowUpsForActiveWorker(t *testing.T) { | |
| 13 | + | if runtime.GOOS == "windows" { | |
| 14 | + | t.Skip("subagent inbox transport uses Unix-domain sockets") | |
| 15 | + | } | |
| 16 | + | ||
| 17 | + | root := shortSocketDir(t) | |
| 18 | + | ready := make(chan struct{}, 1) | |
| 19 | + | commands := make(chan Envelope, 2) | |
| 20 | + | f := New(Config{ | |
| 21 | + | Root: root, RepoRoot: root, | |
| 22 | + | NewRunner: func(a *Agent) Runner { | |
| 23 | + | return RunnerFunc(func(ctx context.Context, _ Sink) error { | |
| 24 | + | listener, err := Listen(a.InboxPath) | |
| 25 | + | if err != nil { | |
| 26 | + | return err | |
| 27 | + | } | |
| 28 | + | defer listener.Close() | |
| 29 | + | a.setProcessState(ProcessAlive) | |
| 30 | + | a.setTurnState(TurnRunning, "turn-1") | |
| 31 | + | ready <- struct{}{} | |
| 32 | + | for { | |
| 33 | + | select { | |
| 34 | + | case <-ctx.Done(): | |
| 35 | + | return ctx.Err() | |
| 36 | + | case line, ok := <-listener.Lines(): | |
| 37 | + | if !ok { | |
| 38 | + | return nil | |
| 39 | + | } | |
| 40 | + | command, err := ParseCommand(line) | |
| 41 | + | if err != nil { | |
| 42 | + | return err | |
| 43 | + | } | |
| 44 | + | if command.Type == CommandAgentShutdown { | |
| 45 | + | return nil | |
| 46 | + | } | |
| 47 | + | commands <- command | |
| 48 | + | } | |
| 49 | + | } | |
| 50 | + | }) | |
| 51 | + | }, | |
| 52 | + | }) | |
| 53 | + | defer f.StopAll() | |
| 54 | + | ||
| 55 | + | a, err := f.Spawn(context.Background(), "review the implementation") | |
| 56 | + | if err != nil { | |
| 57 | + | t.Fatal(err) | |
| 58 | + | } | |
| 59 | + | select { | |
| 60 | + | case <-ready: | |
| 61 | + | case <-time.After(time.Second): | |
| 62 | + | t.Fatal("active worker did not open its inbox") | |
| 63 | + | } | |
| 64 | + | canceled, cancel := context.WithCancel(context.Background()) | |
| 65 | + | cancel() | |
| 66 | + | if _, err := f.ResumeWithPrompt(canceled, a.ID, "must not be accepted"); !errors.Is(err, context.Canceled) { | |
| 67 | + | t.Fatalf("canceled active follow-up error = %v, want context canceled", err) | |
| 68 | + | } | |
| 69 | + | ||
| 70 | + | const ( | |
| 71 | + | firstFollowUp = "Check the parser behavior." | |
| 72 | + | secondFollowUp = "Then check the error handling." | |
| 73 | + | ) | |
| 74 | + | if _, err := f.ResumeWithPrompt(context.Background(), a.ID, firstFollowUp); err != nil { | |
| 75 | + | t.Fatalf("queue first active follow-up: %v", err) | |
| 76 | + | } | |
| 77 | + | if _, err := f.ResumeWithPrompt(context.Background(), a.ID, secondFollowUp); err != nil { | |
| 78 | + | t.Fatalf("queue second active follow-up: %v", err) | |
| 79 | + | } | |
| 80 | + | select { | |
| 81 | + | case command := <-commands: | |
| 82 | + | t.Fatalf("active follow-up was delivered before idle: %v", command.Type) | |
| 83 | + | default: | |
| 84 | + | } | |
| 85 | + | ||
| 86 | + | persisted, err := readAgentMeta(filepath.Join(root, "agents", a.ID)) | |
| 87 | + | if err != nil { | |
| 88 | + | t.Fatal(err) | |
| 89 | + | } | |
| 90 | + | if persisted.ResumePrompt != "" || len(persisted.ResumeQueue) != 2 { | |
| 91 | + | t.Fatalf("active follow-ups persisted as prompt %q queue %d, want empty prompt and queue 2", persisted.ResumePrompt, len(persisted.ResumeQueue)) | |
| 92 | + | } | |
| 93 | + | ||
| 94 | + | updateAgentFromEvent(a, NewEvent(EventAgentIdle, map[string]any{"turn_id": "turn-1"})) | |
| 95 | + | select { | |
| 96 | + | case command := <-commands: | |
| 97 | + | var payload TurnStartPayload | |
| 98 | + | if command.Type != CommandTurnStart || command.DecodePayload(&payload) != nil || payload.Prompt != firstFollowUp { | |
| 99 | + | t.Fatalf("first idle delivery = type %q prompt %q, want turn start %q", command.Type, payload.Prompt, firstFollowUp) | |
| 100 | + | } | |
| 101 | + | if !payload.NewRun { | |
| 102 | + | t.Fatal("first queued follow-up did not request a fresh run budget") | |
| 103 | + | } | |
| 104 | + | case <-time.After(time.Second): | |
| 105 | + | t.Fatal("first active follow-up was not delivered at idle") | |
| 106 | + | } | |
| 107 | + | if got := a.TurnState(); got != TurnQueued { | |
| 108 | + | t.Fatalf("turn state after first queued delivery = %s, want %s", got, TurnQueued) | |
| 109 | + | } | |
| 110 | + | ||
| 111 | + | updateAgentFromEvent(a, NewEvent(EventTurnStarted, map[string]any{ | |
| 112 | + | "turn_id": "turn-2", "step": float64(2), "lifetime_turns": 2, "current_run_turns": 1, | |
| 113 | + | })) | |
| 114 | + | updateAgentFromEvent(a, NewEvent(EventAgentIdle, map[string]any{"turn_id": "turn-2"})) | |
| 115 | + | select { | |
| 116 | + | case command := <-commands: | |
| 117 | + | var payload TurnStartPayload | |
| 118 | + | if command.Type != CommandTurnStart || command.DecodePayload(&payload) != nil || payload.Prompt != secondFollowUp { | |
| 119 | + | t.Fatalf("second idle delivery = type %q prompt %q, want turn start %q", command.Type, payload.Prompt, secondFollowUp) | |
| 120 | + | } | |
| 121 | + | case <-time.After(time.Second): | |
| 122 | + | t.Fatal("second active follow-up was not delivered at the next idle turn") | |
| 123 | + | } | |
| 124 | + | ||
| 125 | + | persisted, err = readAgentMeta(filepath.Join(root, "agents", a.ID)) | |
| 126 | + | if err != nil { | |
| 127 | + | t.Fatal(err) | |
| 128 | + | } | |
| 129 | + | if persisted.ResumePrompt != secondFollowUp || len(persisted.ResumeQueue) != 0 { | |
| 130 | + | t.Fatalf("second follow-up persistence = prompt %q queue %d, want prompt %q and empty queue", persisted.ResumePrompt, len(persisted.ResumeQueue), secondFollowUp) | |
| 131 | + | } | |
| 132 | + | if err := f.Stop(a.ID); err != nil { | |
| 133 | + | t.Fatal(err) | |
| 134 | + | } | |
| 135 | + | a.Wait() | |
| 136 | + | } | |
| 137 | + | ||
| 11 | 138 | func TestResumeWithPromptDeliversFollowUpToIdleWorker(t *testing.T) { | |
| 12 | 139 | if runtime.GOOS == "windows" { | |
| 13 | 140 | t.Skip("subagent inbox transport uses Unix-domain sockets") | |
| @@ -60,6 +187,7 @@ func TestResumeWithPromptDeliversFollowUpToIdleWorker(t *testing.T) { | |||
| 60 | 187 | case <-time.After(time.Second): | |
| 61 | 188 | t.Fatal("idle worker did not open its inbox") | |
| 62 | 189 | } | |
| 190 | + | a.setTurnCounts(4, 2) | |
| 63 | 191 | ||
| 64 | 192 | const followUp = "I applied your review. What do you think now?" | |
| 65 | 193 | continued, err := f.ResumeWithPrompt(context.Background(), a.ID, followUp) | |
| @@ -72,6 +200,12 @@ func TestResumeWithPromptDeliversFollowUpToIdleWorker(t *testing.T) { | |||
| 72 | 200 | if got := a.TurnState(); got != TurnQueued { | |
| 73 | 201 | t.Fatalf("turn state after sending follow-up = %s, want %s", got, TurnQueued) | |
| 74 | 202 | } | |
| 203 | + | if got := a.CurrentRunTurnsValue(); got != 0 { | |
| 204 | + | t.Fatalf("current-run turns after explicit resume = %d, want 0 before turn starts", got) | |
| 205 | + | } | |
| 206 | + | if got := a.LifetimeTurnsValue(); got != 4 { | |
| 207 | + | t.Fatalf("lifetime turns after explicit resume = %d, want 4", got) | |
| 208 | + | } | |
| 75 | 209 | if _, err := f.ResumeWithPrompt(context.Background(), a.ID, "duplicate follow-up"); err == nil { | |
| 76 | 210 | t.Fatal("second follow-up succeeded while the first was queued") | |
| 77 | 211 | } | |
| @@ -87,6 +221,9 @@ func TestResumeWithPromptDeliversFollowUpToIdleWorker(t *testing.T) { | |||
| 87 | 221 | if payload.Prompt != followUp { | |
| 88 | 222 | t.Fatalf("follow-up prompt = %q, want %q", payload.Prompt, followUp) | |
| 89 | 223 | } | |
| 224 | + | if !payload.NewRun { | |
| 225 | + | t.Fatal("idle follow-up did not request a fresh run budget") | |
| 226 | + | } | |
| 90 | 227 | case <-time.After(time.Second): | |
| 91 | 228 | t.Fatal("idle worker did not receive the follow-up") | |
| 92 | 229 | } | |
| @@ -2,12 +2,139 @@ package subagents | |||||
| 2 | 2 | ||||
| 3 | import ( | 3 | import ( | ||
| 4 | "context" | 4 | "context" | ||
| 5 | + | "errors" | |||
| 5 | "path/filepath" | 6 | "path/filepath" | ||
| 6 | "runtime" | 7 | "runtime" | ||
| 7 | "testing" | 8 | "testing" | ||
| 8 | "time" | 9 | "time" | ||
| 9 | ) | 10 | ) | ||
| 10 | 11 | ||||
| 12 | + | func TestResumeWithPromptQueuesFollowUpsForActiveWorker(t *testing.T) { | |||
| 13 | + | if runtime.GOOS == "windows" { | |||
| 14 | + | t.Skip("subagent inbox transport uses Unix-domain sockets") | |||
| 15 | + | } | |||
| 16 | + | ||||
| 17 | + | root := shortSocketDir(t) | |||
| 18 | + | ready := make(chan struct{}, 1) | |||
| 19 | + | commands := make(chan Envelope, 2) | |||
| 20 | + | f := New(Config{ | |||
| 21 | + | Root: root, RepoRoot: root, | |||
| 22 | + | NewRunner: func(a *Agent) Runner { | |||
| 23 | + | return RunnerFunc(func(ctx context.Context, _ Sink) error { | |||
| 24 | + | listener, err := Listen(a.InboxPath) | |||
| 25 | + | if err != nil { | |||
| 26 | + | return err | |||
| 27 | + | } | |||
| 28 | + | defer listener.Close() | |||
| 29 | + | a.setProcessState(ProcessAlive) | |||
| 30 | + | a.setTurnState(TurnRunning, "turn-1") | |||
| 31 | + | ready <- struct{}{} | |||
| 32 | + | for { | |||
| 33 | + | select { | |||
| 34 | + | case <-ctx.Done(): | |||
| 35 | + | return ctx.Err() | |||
| 36 | + | case line, ok := <-listener.Lines(): | |||
| 37 | + | if !ok { | |||
| 38 | + | return nil | |||
| 39 | + | } | |||
| 40 | + | command, err := ParseCommand(line) | |||
| 41 | + | if err != nil { | |||
| 42 | + | return err | |||
| 43 | + | } | |||
| 44 | + | if command.Type == CommandAgentShutdown { | |||
| 45 | + | return nil | |||
| 46 | + | } | |||
| 47 | + | commands <- command | |||
| 48 | + | } | |||
| 49 | + | } | |||
| 50 | + | }) | |||
| 51 | + | }, | |||
| 52 | + | }) | |||
| 53 | + | defer f.StopAll() | |||
| 54 | + | ||||
| 55 | + | a, err := f.Spawn(context.Background(), "review the implementation") | |||
| 56 | + | if err != nil { | |||
| 57 | + | t.Fatal(err) | |||
| 58 | + | } | |||
| 59 | + | select { | |||
| 60 | + | case <-ready: | |||
| 61 | + | case <-time.After(time.Second): | |||
| 62 | + | t.Fatal("active worker did not open its inbox") | |||
| 63 | + | } | |||
| 64 | + | canceled, cancel := context.WithCancel(context.Background()) | |||
| 65 | + | cancel() | |||
| 66 | + | if _, err := f.ResumeWithPrompt(canceled, a.ID, "must not be accepted"); !errors.Is(err, context.Canceled) { | |||
| 67 | + | t.Fatalf("canceled active follow-up error = %v, want context canceled", err) | |||
| 68 | + | } | |||
| 69 | + | ||||
| 70 | + | const ( | |||
| 71 | + | firstFollowUp = "Check the parser behavior." | |||
| 72 | + | secondFollowUp = "Then check the error handling." | |||
| 73 | + | ) | |||
| 74 | + | if _, err := f.ResumeWithPrompt(context.Background(), a.ID, firstFollowUp); err != nil { | |||
| 75 | + | t.Fatalf("queue first active follow-up: %v", err) | |||
| 76 | + | } | |||
| 77 | + | if _, err := f.ResumeWithPrompt(context.Background(), a.ID, secondFollowUp); err != nil { | |||
| 78 | + | t.Fatalf("queue second active follow-up: %v", err) | |||
| 79 | + | } | |||
| 80 | + | select { | |||
| 81 | + | case command := <-commands: | |||
| 82 | + | t.Fatalf("active follow-up was delivered before idle: %v", command.Type) | |||
| 83 | + | default: | |||
| 84 | + | } | |||
| 85 | + | ||||
| 86 | + | persisted, err := readAgentMeta(filepath.Join(root, "agents", a.ID)) | |||
| 87 | + | if err != nil { | |||
| 88 | + | t.Fatal(err) | |||
| 89 | + | } | |||
| 90 | + | if persisted.ResumePrompt != "" || len(persisted.ResumeQueue) != 2 { | |||
| 91 | + | t.Fatalf("active follow-ups persisted as prompt %q queue %d, want empty prompt and queue 2", persisted.ResumePrompt, len(persisted.ResumeQueue)) | |||
| 92 | + | } | |||
| 93 | + | ||||
| 94 | + | updateAgentFromEvent(a, NewEvent(EventAgentIdle, map[string]any{"turn_id": "turn-1"})) | |||
| 95 | + | select { | |||
| 96 | + | case command := <-commands: | |||
| 97 | + | var payload TurnStartPayload | |||
| 98 | + | if command.Type != CommandTurnStart || command.DecodePayload(&payload) != nil || payload.Prompt != firstFollowUp { | |||
| 99 | + | t.Fatalf("first idle delivery = type %q prompt %q, want turn start %q", command.Type, payload.Prompt, firstFollowUp) | |||
| 100 | + | } | |||
| 101 | + | if !payload.NewRun { | |||
| 102 | + | t.Fatal("first queued follow-up did not request a fresh run budget") | |||
| 103 | + | } | |||
| 104 | + | case <-time.After(time.Second): | |||
| 105 | + | t.Fatal("first active follow-up was not delivered at idle") | |||
| 106 | + | } | |||
| 107 | + | if got := a.TurnState(); got != TurnQueued { | |||
| 108 | + | t.Fatalf("turn state after first queued delivery = %s, want %s", got, TurnQueued) | |||
| 109 | + | } | |||
| 110 | + | ||||
| 111 | + | updateAgentFromEvent(a, NewEvent(EventTurnStarted, map[string]any{ | |||
| 112 | + | "turn_id": "turn-2", "step": float64(2), "lifetime_turns": 2, "current_run_turns": 1, | |||
| 113 | + | })) | |||
| 114 | + | updateAgentFromEvent(a, NewEvent(EventAgentIdle, map[string]any{"turn_id": "turn-2"})) | |||
| 115 | + | select { | |||
| 116 | + | case command := <-commands: | |||
| 117 | + | var payload TurnStartPayload | |||
| 118 | + | if command.Type != CommandTurnStart || command.DecodePayload(&payload) != nil || payload.Prompt != secondFollowUp { | |||
| 119 | + | t.Fatalf("second idle delivery = type %q prompt %q, want turn start %q", command.Type, payload.Prompt, secondFollowUp) | |||
| 120 | + | } | |||
| 121 | + | case <-time.After(time.Second): | |||
| 122 | + | t.Fatal("second active follow-up was not delivered at the next idle turn") | |||
| 123 | + | } | |||
| 124 | + | ||||
| 125 | + | persisted, err = readAgentMeta(filepath.Join(root, "agents", a.ID)) | |||
| 126 | + | if err != nil { | |||
| 127 | + | t.Fatal(err) | |||
| 128 | + | } | |||
| 129 | + | if persisted.ResumePrompt != secondFollowUp || len(persisted.ResumeQueue) != 0 { | |||
| 130 | + | t.Fatalf("second follow-up persistence = prompt %q queue %d, want prompt %q and empty queue", persisted.ResumePrompt, len(persisted.ResumeQueue), secondFollowUp) | |||
| 131 | + | } | |||
| 132 | + | if err := f.Stop(a.ID); err != nil { | |||
| 133 | + | t.Fatal(err) | |||
| 134 | + | } | |||
| 135 | + | a.Wait() | |||
| 136 | + | } | |||
| 137 | + | ||||
| 11 | func TestResumeWithPromptDeliversFollowUpToIdleWorker(t *testing.T) { | 138 | func TestResumeWithPromptDeliversFollowUpToIdleWorker(t *testing.T) { | ||
| 12 | if runtime.GOOS == "windows" { | 139 | if runtime.GOOS == "windows" { | ||
| 13 | t.Skip("subagent inbox transport uses Unix-domain sockets") | 140 | t.Skip("subagent inbox transport uses Unix-domain sockets") | ||
| @@ -60,6 +187,7 @@ func TestResumeWithPromptDeliversFollowUpToIdleWorker(t *testing.T) { | |||||
| 60 | case <-time.After(time.Second): | 187 | case <-time.After(time.Second): | ||
| 61 | t.Fatal("idle worker did not open its inbox") | 188 | t.Fatal("idle worker did not open its inbox") | ||
| 62 | } | 189 | } | ||
| 190 | + | a.setTurnCounts(4, 2) | |||
| 63 | 191 | ||||
| 64 | const followUp = "I applied your review. What do you think now?" | 192 | const followUp = "I applied your review. What do you think now?" | ||
| 65 | continued, err := f.ResumeWithPrompt(context.Background(), a.ID, followUp) | 193 | continued, err := f.ResumeWithPrompt(context.Background(), a.ID, followUp) | ||
| @@ -72,6 +200,12 @@ func TestResumeWithPromptDeliversFollowUpToIdleWorker(t *testing.T) { | |||||
| 72 | if got := a.TurnState(); got != TurnQueued { | 200 | if got := a.TurnState(); got != TurnQueued { | ||
| 73 | t.Fatalf("turn state after sending follow-up = %s, want %s", got, TurnQueued) | 201 | t.Fatalf("turn state after sending follow-up = %s, want %s", got, TurnQueued) | ||
| 74 | } | 202 | } | ||
| 203 | + | if got := a.CurrentRunTurnsValue(); got != 0 { | |||
| 204 | + | t.Fatalf("current-run turns after explicit resume = %d, want 0 before turn starts", got) | |||
| 205 | + | } | |||
| 206 | + | if got := a.LifetimeTurnsValue(); got != 4 { | |||
| 207 | + | t.Fatalf("lifetime turns after explicit resume = %d, want 4", got) | |||
| 208 | + | } | |||
| 75 | if _, err := f.ResumeWithPrompt(context.Background(), a.ID, "duplicate follow-up"); err == nil { | 209 | if _, err := f.ResumeWithPrompt(context.Background(), a.ID, "duplicate follow-up"); err == nil { | ||
| 76 | t.Fatal("second follow-up succeeded while the first was queued") | 210 | t.Fatal("second follow-up succeeded while the first was queued") | ||
| 77 | } | 211 | } | ||
| @@ -87,6 +221,9 @@ func TestResumeWithPromptDeliversFollowUpToIdleWorker(t *testing.T) { | |||||
| 87 | if payload.Prompt != followUp { | 221 | if payload.Prompt != followUp { | ||
| 88 | t.Fatalf("follow-up prompt = %q, want %q", payload.Prompt, followUp) | 222 | t.Fatalf("follow-up prompt = %q, want %q", payload.Prompt, followUp) | ||
| 89 | } | 223 | } | ||
| 224 | + | if !payload.NewRun { | |||
| 225 | + | t.Fatal("idle follow-up did not request a fresh run budget") | |||
| 226 | + | } | |||
| 90 | case <-time.After(time.Second): | 227 | case <-time.After(time.Second): | ||
| 91 | t.Fatal("idle worker did not receive the follow-up") | 228 | t.Fatal("idle worker did not receive the follow-up") | ||
| 92 | } | 229 | } | ||
M packages/agent/subagents/runner.go +120 -14
| @@ -73,6 +73,9 @@ type subagentWorkerArgsOpts struct { | |||
| 73 | 73 | FastModeSet bool | |
| 74 | 74 | Subagent string | |
| 75 | 75 | MaxTurns int | |
| 76 | + | LifetimeTurns int | |
| 77 | + | RunTurns int | |
| 78 | + | CountersSet bool | |
| 76 | 79 | Tools []string | |
| 77 | 80 | WebSearchPolicy WebSearchPolicy | |
| 78 | 81 | } | |
| @@ -108,6 +111,9 @@ func defaultChildArgs(exe string, a *Agent, sessionPath, inboxPath string) []str | |||
| 108 | 111 | FastModeSet: true, | |
| 109 | 112 | Subagent: a.Subagent, | |
| 110 | 113 | MaxTurns: a.MaxTurns, | |
| 114 | + | LifetimeTurns: a.LifetimeTurnsValue(), | |
| 115 | + | RunTurns: a.CurrentRunTurnsValue(), | |
| 116 | + | CountersSet: true, | |
| 111 | 117 | Tools: a.Tools, | |
| 112 | 118 | WebSearchPolicy: a.WebSearchPolicy, | |
| 113 | 119 | }) | |
| @@ -151,6 +157,12 @@ func subagentWorkerArgs(opts subagentWorkerArgsOpts) []string { | |||
| 151 | 157 | if opts.MaxTurns > 0 { | |
| 152 | 158 | args = append(args, "--max-turns", fmt.Sprint(opts.MaxTurns)) | |
| 153 | 159 | } | |
| 160 | + | if opts.CountersSet { | |
| 161 | + | args = append(args, | |
| 162 | + | "--subagent-lifetime-turns", fmt.Sprint(opts.LifetimeTurns), | |
| 163 | + | "--subagent-run-turns", fmt.Sprint(opts.RunTurns), | |
| 164 | + | ) | |
| 165 | + | } | |
| 154 | 166 | webSearchPolicy := childWebSearchPolicy(opts.WebSearchPolicy, opts.Subagent, opts.Tools) | |
| 155 | 167 | // Always propagate the final capability decision, including deny, so a | |
| 156 | 168 | // child with no ordinary --tools list cannot re-enable web search from its | |
| @@ -397,7 +409,9 @@ func (r *execRunner) Run(ctx context.Context, sink Sink) error { | |||
| 397 | 409 | if trimmed != "" && !truncated { | |
| 398 | 410 | if ev, ok := parseEventLine(trimmed); ok { | |
| 399 | 411 | appendLog(ev) | |
| 400 | - | updateAgentFromEvent(r.agent, ev) | |
| 412 | + | if persistErr := updateAgentFromEvent(r.agent, ev); persistErr != nil { | |
| 413 | + | sink.Transcript("error: metadata persistence failed: " + persistErr.Error()) | |
| 414 | + | } | |
| 401 | 415 | applyEventToSink(ev, sink) | |
| 402 | 416 | // Fan prompt-level task completions up to any | |
| 403 | 417 | // subscriber on the supervised Agent. The child | |
| @@ -594,9 +608,41 @@ func parseEventLine(line string) (Event, bool) { | |||
| 594 | 608 | }, true | |
| 595 | 609 | } | |
| 596 | 610 | ||
| 597 | - | func updateAgentFromEvent(a *Agent, ev Event) { | |
| 611 | + | func eventMatchesPendingResume(a *Agent, ev Event) bool { | |
| 598 | 612 | if a == nil { | |
| 599 | - | return | |
| 613 | + | return false | |
| 614 | + | } | |
| 615 | + | commandID, _ := ev.Data["command_id"].(string) | |
| 616 | + | if commandID == "" { | |
| 617 | + | return true // protocol-v1 events predate command identities | |
| 618 | + | } | |
| 619 | + | expected := a.resumePromptCommandID() | |
| 620 | + | return expected == "" || expected == commandID | |
| 621 | + | } | |
| 622 | + | ||
| 623 | + | func eventCounter(data map[string]any, key string) (int, bool) { | |
| 624 | + | value, ok := data[key] | |
| 625 | + | if !ok { | |
| 626 | + | return 0, false | |
| 627 | + | } | |
| 628 | + | switch value := value.(type) { | |
| 629 | + | case int: | |
| 630 | + | return value, true | |
| 631 | + | case int64: | |
| 632 | + | return int(value), true | |
| 633 | + | case float64: | |
| 634 | + | return int(value), value == float64(int(value)) | |
| 635 | + | case json.Number: | |
| 636 | + | parsed, err := value.Int64() | |
| 637 | + | return int(parsed), err == nil | |
| 638 | + | default: | |
| 639 | + | return 0, false | |
| 640 | + | } | |
| 641 | + | } | |
| 642 | + | ||
| 643 | + | func updateAgentFromEvent(a *Agent, ev Event) error { | |
| 644 | + | if a == nil { | |
| 645 | + | return nil | |
| 600 | 646 | } | |
| 601 | 647 | now := ev.Time | |
| 602 | 648 | if now.IsZero() { | |
| @@ -604,25 +650,56 @@ func updateAgentFromEvent(a *Agent, ev Event) { | |||
| 604 | 650 | } | |
| 605 | 651 | a.markActivity(now) | |
| 606 | 652 | persist := false | |
| 653 | + | notifyIdle := false | |
| 607 | 654 | switch ev.Type { | |
| 608 | 655 | case EventAgentReady, "agent_ready": | |
| 609 | 656 | a.setProcessState(ProcessAlive) | |
| 610 | - | a.setTurnState(TurnIdle, ev.TurnID) | |
| 657 | + | if lifetime, ok := eventCounter(ev.Data, "lifetime_turns"); ok { | |
| 658 | + | if currentRun, currentOK := eventCounter(ev.Data, "current_run_turns"); currentOK { | |
| 659 | + | a.setTurnCounts(lifetime, currentRun) | |
| 660 | + | } | |
| 661 | + | } | |
| 662 | + | // A resumed worker may have a durable initial follow-up in its argv. | |
| 663 | + | // Keep it queued until the worker emits its own turn.started event; | |
| 664 | + | // treating readiness as idle here could dispatch another queued prompt | |
| 665 | + | // concurrently with that initial turn. | |
| 666 | + | if prompt, _ := a.ResumePromptInfo(); prompt == "" && a.TurnState() != TurnQueued { | |
| 667 | + | a.setTurnState(TurnIdle, ev.TurnID) | |
| 668 | + | notifyIdle = true | |
| 669 | + | } | |
| 611 | 670 | persist = true | |
| 612 | 671 | case EventAgentHeartbeat, "agent_heartbeat": | |
| 613 | 672 | a.setProcessState(ProcessAlive) | |
| 614 | 673 | persist = true | |
| 615 | - | case EventTurnStarted, "turn_start": | |
| 616 | - | turnID := ev.TurnID | |
| 617 | - | if turnID == "" { | |
| 618 | - | if step, ok := ev.Data["step"].(float64); ok { | |
| 619 | - | turnID = fmt.Sprintf("turn-%d", int(step)) | |
| 620 | - | } | |
| 674 | + | case EventTurnStarted: | |
| 675 | + | if !isDelegatedTurnStart(ev) { | |
| 676 | + | // Provider/model-loop turn starts are nested inside one worker | |
| 677 | + | // message turn and are activity only. | |
| 678 | + | a.setProcessState(ProcessAlive) | |
| 679 | + | break | |
| 621 | 680 | } | |
| 681 | + | if !eventMatchesPendingResume(a, ev) { | |
| 682 | + | break | |
| 683 | + | } | |
| 684 | + | turnID := eventTurnID(ev) | |
| 622 | 685 | a.setProcessState(ProcessAlive) | |
| 686 | + | if lifetime, ok := eventCounter(ev.Data, "lifetime_turns"); ok { | |
| 687 | + | if currentRun, currentOK := eventCounter(ev.Data, "current_run_turns"); currentOK { | |
| 688 | + | a.setTurnCounts(lifetime, currentRun) | |
| 689 | + | } else { | |
| 690 | + | a.incrementTurnCounts() | |
| 691 | + | } | |
| 692 | + | } else { | |
| 693 | + | a.incrementTurnCounts() | |
| 694 | + | } | |
| 623 | 695 | a.setTurnState(TurnRunning, turnID) | |
| 624 | 696 | a.clearResumePrompt() | |
| 625 | 697 | persist = true | |
| 698 | + | case "turn_start": | |
| 699 | + | // Provider/model-loop turn starts are nested inside one worker message | |
| 700 | + | // turn. They are activity only and must not consume or reset the | |
| 701 | + | // worker's lifetime or current-run budget. | |
| 702 | + | a.setProcessState(ProcessAlive) | |
| 626 | 703 | case EventTurnResult, "turn_result": | |
| 627 | 704 | if result, err := decodeTurnResultEvent(ev, a.ID, a.maxOutputBytes, a.maxOutputLines); err == nil { | |
| 628 | 705 | a.setResult(result) | |
| @@ -653,17 +730,36 @@ func updateAgentFromEvent(a *Agent, ev Event) { | |||
| 653 | 730 | a.setTurnState(TurnFailed, ev.TurnID) | |
| 654 | 731 | persist = true | |
| 655 | 732 | case EventAgentIdle, "agent_idle": | |
| 733 | + | if lifetime, ok := eventCounter(ev.Data, "lifetime_turns"); ok { | |
| 734 | + | if currentRun, currentOK := eventCounter(ev.Data, "current_run_turns"); currentOK { | |
| 735 | + | a.setTurnCounts(lifetime, currentRun) | |
| 736 | + | } | |
| 737 | + | } | |
| 656 | 738 | a.setTurnState(TurnIdle, ev.TurnID) | |
| 739 | + | notifyIdle = true | |
| 657 | 740 | persist = true | |
| 658 | 741 | case EventAgentExited, "agent_exited", "agent_stopped": | |
| 659 | 742 | a.setProcessState(ProcessExited) | |
| 660 | 743 | persist = true | |
| 744 | + | case "error": | |
| 745 | + | if code, _ := ev.Data["code"].(string); code == "turn_rejected" && eventMatchesPendingResume(a, ev) { | |
| 746 | + | if a.rejectActiveResumePrompt() { | |
| 747 | + | reason, _ := ev.Data["reason"].(string) | |
| 748 | + | if reason == "max_turns" { | |
| 749 | + | a.setTurnState(TurnFailed, ev.TurnID) | |
| 750 | + | } else { | |
| 751 | + | a.setTurnState(TurnIdle, ev.TurnID) | |
| 752 | + | notifyIdle = true | |
| 753 | + | } | |
| 754 | + | persist = true | |
| 755 | + | } | |
| 756 | + | } | |
| 661 | 757 | case "turn_end": | |
| 662 | 758 | // Provider/tool-loop turn_end events (for example stop=tool_use) | |
| 663 | 759 | // do not carry the daemon's prompt step and are not terminal for | |
| 664 | 760 | // the delegated task. They remain in the event log, but must not | |
| 665 | 761 | // overwrite the delegated turn state or trigger persistence. | |
| 666 | - | if _, ok := ev.Data["step"].(float64); !ok { | |
| 762 | + | if !isDelegatedTurnEnd(ev) { | |
| 667 | 763 | break | |
| 668 | 764 | } | |
| 669 | 765 | if message, _ := ev.Data["error"].(string); message != "" { | |
| @@ -674,8 +770,15 @@ func updateAgentFromEvent(a *Agent, ev Event) { | |||
| 674 | 770 | persist = true | |
| 675 | 771 | } | |
| 676 | 772 | if persist && a.persistFn != nil { | |
| 677 | - | a.persistFn(a) | |
| 773 | + | if err := a.persistFn(a); err != nil { | |
| 774 | + | a.recordPersistenceError(err) | |
| 775 | + | return err | |
| 776 | + | } | |
| 678 | 777 | } | |
| 778 | + | if notifyIdle { | |
| 779 | + | a.notifyTurnIdle() | |
| 780 | + | } | |
| 781 | + | return nil | |
| 679 | 782 | } | |
| 680 | 783 | ||
| 681 | 784 | // notifyPromptTurnEnd calls Agent.OnTurnEnd only for the subagent | |
| @@ -686,7 +789,10 @@ func notifyPromptTurnEnd(a *Agent, ev Event) { | |||
| 686 | 789 | if a == nil || ev.Type != "turn_end" { | |
| 687 | 790 | return | |
| 688 | 791 | } | |
| 689 | - | step, ok := ev.Data["step"].(float64) | |
| 792 | + | if !isDelegatedTurnEnd(ev) { | |
| 793 | + | return | |
| 794 | + | } | |
| 795 | + | step, ok := eventCounter(ev.Data, "step") | |
| 690 | 796 | if !ok { | |
| 691 | 797 | return | |
| 692 | 798 | } | |
| @@ -700,7 +806,7 @@ func notifyPromptTurnEnd(a *Agent, ev Event) { | |||
| 700 | 806 | return | |
| 701 | 807 | } | |
| 702 | 808 | a.mu.Unlock() | |
| 703 | - | go fn(int(step), errMsg) | |
| 809 | + | go fn(step, errMsg) | |
| 704 | 810 | } | |
| 705 | 811 | ||
| 706 | 812 | // applyEventToSink translates an Event into Sink updates. Only a | |
| @@ -73,6 +73,9 @@ type subagentWorkerArgsOpts struct { | |||||
| 73 | FastModeSet bool | 73 | FastModeSet bool | ||
| 74 | Subagent string | 74 | Subagent string | ||
| 75 | MaxTurns int | 75 | MaxTurns int | ||
| 76 | + | LifetimeTurns int | |||
| 77 | + | RunTurns int | |||
| 78 | + | CountersSet bool | |||
| 76 | Tools []string | 79 | Tools []string | ||
| 77 | WebSearchPolicy WebSearchPolicy | 80 | WebSearchPolicy WebSearchPolicy | ||
| 78 | } | 81 | } | ||
| @@ -108,6 +111,9 @@ func defaultChildArgs(exe string, a *Agent, sessionPath, inboxPath string) []str | |||||
| 108 | FastModeSet: true, | 111 | FastModeSet: true, | ||
| 109 | Subagent: a.Subagent, | 112 | Subagent: a.Subagent, | ||
| 110 | MaxTurns: a.MaxTurns, | 113 | MaxTurns: a.MaxTurns, | ||
| 114 | + | LifetimeTurns: a.LifetimeTurnsValue(), | |||
| 115 | + | RunTurns: a.CurrentRunTurnsValue(), | |||
| 116 | + | CountersSet: true, | |||
| 111 | Tools: a.Tools, | 117 | Tools: a.Tools, | ||
| 112 | WebSearchPolicy: a.WebSearchPolicy, | 118 | WebSearchPolicy: a.WebSearchPolicy, | ||
| 113 | }) | 119 | }) | ||
| @@ -151,6 +157,12 @@ func subagentWorkerArgs(opts subagentWorkerArgsOpts) []string { | |||||
| 151 | if opts.MaxTurns > 0 { | 157 | if opts.MaxTurns > 0 { | ||
| 152 | args = append(args, "--max-turns", fmt.Sprint(opts.MaxTurns)) | 158 | args = append(args, "--max-turns", fmt.Sprint(opts.MaxTurns)) | ||
| 153 | } | 159 | } | ||
| 160 | + | if opts.CountersSet { | |||
| 161 | + | args = append(args, | |||
| 162 | + | "--subagent-lifetime-turns", fmt.Sprint(opts.LifetimeTurns), | |||
| 163 | + | "--subagent-run-turns", fmt.Sprint(opts.RunTurns), | |||
| 164 | + | ) | |||
| 165 | + | } | |||
| 154 | webSearchPolicy := childWebSearchPolicy(opts.WebSearchPolicy, opts.Subagent, opts.Tools) | 166 | webSearchPolicy := childWebSearchPolicy(opts.WebSearchPolicy, opts.Subagent, opts.Tools) | ||
| 155 | // Always propagate the final capability decision, including deny, so a | 167 | // Always propagate the final capability decision, including deny, so a | ||
| 156 | // child with no ordinary --tools list cannot re-enable web search from its | 168 | // child with no ordinary --tools list cannot re-enable web search from its | ||
| @@ -397,7 +409,9 @@ func (r *execRunner) Run(ctx context.Context, sink Sink) error { | |||||
| 397 | if trimmed != "" && !truncated { | 409 | if trimmed != "" && !truncated { | ||
| 398 | if ev, ok := parseEventLine(trimmed); ok { | 410 | if ev, ok := parseEventLine(trimmed); ok { | ||
| 399 | appendLog(ev) | 411 | appendLog(ev) | ||
| 400 | - | updateAgentFromEvent(r.agent, ev) | 412 | + | if persistErr := updateAgentFromEvent(r.agent, ev); persistErr != nil { |
| 413 | + | sink.Transcript("error: metadata persistence failed: " + persistErr.Error()) | |||
| 414 | + | } | |||
| 401 | applyEventToSink(ev, sink) | 415 | applyEventToSink(ev, sink) | ||
| 402 | // Fan prompt-level task completions up to any | 416 | // Fan prompt-level task completions up to any | ||
| 403 | // subscriber on the supervised Agent. The child | 417 | // subscriber on the supervised Agent. The child | ||
| @@ -594,9 +608,41 @@ func parseEventLine(line string) (Event, bool) { | |||||
| 594 | }, true | 608 | }, true | ||
| 595 | } | 609 | } | ||
| 596 | 610 | ||||
| 597 | - | func updateAgentFromEvent(a *Agent, ev Event) { | 611 | + | func eventMatchesPendingResume(a *Agent, ev Event) bool { |
| 598 | if a == nil { | 612 | if a == nil { | ||
| 599 | - | return | 613 | + | return false |
| 614 | + | } | |||
| 615 | + | commandID, _ := ev.Data["command_id"].(string) | |||
| 616 | + | if commandID == "" { | |||
| 617 | + | return true // protocol-v1 events predate command identities | |||
| 618 | + | } | |||
| 619 | + | expected := a.resumePromptCommandID() | |||
| 620 | + | return expected == "" || expected == commandID | |||
| 621 | + | } | |||
| 622 | + | ||||
| 623 | + | func eventCounter(data map[string]any, key string) (int, bool) { | |||
| 624 | + | value, ok := data[key] | |||
| 625 | + | if !ok { | |||
| 626 | + | return 0, false | |||
| 627 | + | } | |||
| 628 | + | switch value := value.(type) { | |||
| 629 | + | case int: | |||
| 630 | + | return value, true | |||
| 631 | + | case int64: | |||
| 632 | + | return int(value), true | |||
| 633 | + | case float64: | |||
| 634 | + | return int(value), value == float64(int(value)) | |||
| 635 | + | case json.Number: | |||
| 636 | + | parsed, err := value.Int64() | |||
| 637 | + | return int(parsed), err == nil | |||
| 638 | + | default: | |||
| 639 | + | return 0, false | |||
| 640 | + | } | |||
| 641 | + | } | |||
| 642 | + | ||||
| 643 | + | func updateAgentFromEvent(a *Agent, ev Event) error { | |||
| 644 | + | if a == nil { | |||
| 645 | + | return nil | |||
| 600 | } | 646 | } | ||
| 601 | now := ev.Time | 647 | now := ev.Time | ||
| 602 | if now.IsZero() { | 648 | if now.IsZero() { | ||
| @@ -604,25 +650,56 @@ func updateAgentFromEvent(a *Agent, ev Event) { | |||||
| 604 | } | 650 | } | ||
| 605 | a.markActivity(now) | 651 | a.markActivity(now) | ||
| 606 | persist := false | 652 | persist := false | ||
| 653 | + | notifyIdle := false | |||
| 607 | switch ev.Type { | 654 | switch ev.Type { | ||
| 608 | case EventAgentReady, "agent_ready": | 655 | case EventAgentReady, "agent_ready": | ||
| 609 | a.setProcessState(ProcessAlive) | 656 | a.setProcessState(ProcessAlive) | ||
| 610 | - | a.setTurnState(TurnIdle, ev.TurnID) | 657 | + | if lifetime, ok := eventCounter(ev.Data, "lifetime_turns"); ok { |
| 658 | + | if currentRun, currentOK := eventCounter(ev.Data, "current_run_turns"); currentOK { | |||
| 659 | + | a.setTurnCounts(lifetime, currentRun) | |||
| 660 | + | } | |||
| 661 | + | } | |||
| 662 | + | // A resumed worker may have a durable initial follow-up in its argv. | |||
| 663 | + | // Keep it queued until the worker emits its own turn.started event; | |||
| 664 | + | // treating readiness as idle here could dispatch another queued prompt | |||
| 665 | + | // concurrently with that initial turn. | |||
| 666 | + | if prompt, _ := a.ResumePromptInfo(); prompt == "" && a.TurnState() != TurnQueued { | |||
| 667 | + | a.setTurnState(TurnIdle, ev.TurnID) | |||
| 668 | + | notifyIdle = true | |||
| 669 | + | } | |||
| 611 | persist = true | 670 | persist = true | ||
| 612 | case EventAgentHeartbeat, "agent_heartbeat": | 671 | case EventAgentHeartbeat, "agent_heartbeat": | ||
| 613 | a.setProcessState(ProcessAlive) | 672 | a.setProcessState(ProcessAlive) | ||
| 614 | persist = true | 673 | persist = true | ||
| 615 | - | case EventTurnStarted, "turn_start": | 674 | + | case EventTurnStarted: |
| 616 | - | turnID := ev.TurnID | 675 | + | if !isDelegatedTurnStart(ev) { |
| 617 | - | if turnID == "" { | 676 | + | // Provider/model-loop turn starts are nested inside one worker |
| 618 | - | if step, ok := ev.Data["step"].(float64); ok { | 677 | + | // message turn and are activity only. |
| 619 | - | turnID = fmt.Sprintf("turn-%d", int(step)) | 678 | + | a.setProcessState(ProcessAlive) |
| 620 | - | } | 679 | + | break |
| 621 | } | 680 | } | ||
| 681 | + | if !eventMatchesPendingResume(a, ev) { | |||
| 682 | + | break | |||
| 683 | + | } | |||
| 684 | + | turnID := eventTurnID(ev) | |||
| 622 | a.setProcessState(ProcessAlive) | 685 | a.setProcessState(ProcessAlive) | ||
| 686 | + | if lifetime, ok := eventCounter(ev.Data, "lifetime_turns"); ok { | |||
| 687 | + | if currentRun, currentOK := eventCounter(ev.Data, "current_run_turns"); currentOK { | |||
| 688 | + | a.setTurnCounts(lifetime, currentRun) | |||
| 689 | + | } else { | |||
| 690 | + | a.incrementTurnCounts() | |||
| 691 | + | } | |||
| 692 | + | } else { | |||
| 693 | + | a.incrementTurnCounts() | |||
| 694 | + | } | |||
| 623 | a.setTurnState(TurnRunning, turnID) | 695 | a.setTurnState(TurnRunning, turnID) | ||
| 624 | a.clearResumePrompt() | 696 | a.clearResumePrompt() | ||
| 625 | persist = true | 697 | persist = true | ||
| 698 | + | case "turn_start": | |||
| 699 | + | // Provider/model-loop turn starts are nested inside one worker message | |||
| 700 | + | // turn. They are activity only and must not consume or reset the | |||
| 701 | + | // worker's lifetime or current-run budget. | |||
| 702 | + | a.setProcessState(ProcessAlive) | |||
| 626 | case EventTurnResult, "turn_result": | 703 | case EventTurnResult, "turn_result": | ||
| 627 | if result, err := decodeTurnResultEvent(ev, a.ID, a.maxOutputBytes, a.maxOutputLines); err == nil { | 704 | if result, err := decodeTurnResultEvent(ev, a.ID, a.maxOutputBytes, a.maxOutputLines); err == nil { | ||
| 628 | a.setResult(result) | 705 | a.setResult(result) | ||
| @@ -653,17 +730,36 @@ func updateAgentFromEvent(a *Agent, ev Event) { | |||||
| 653 | a.setTurnState(TurnFailed, ev.TurnID) | 730 | a.setTurnState(TurnFailed, ev.TurnID) | ||
| 654 | persist = true | 731 | persist = true | ||
| 655 | case EventAgentIdle, "agent_idle": | 732 | case EventAgentIdle, "agent_idle": | ||
| 733 | + | if lifetime, ok := eventCounter(ev.Data, "lifetime_turns"); ok { | |||
| 734 | + | if currentRun, currentOK := eventCounter(ev.Data, "current_run_turns"); currentOK { | |||
| 735 | + | a.setTurnCounts(lifetime, currentRun) | |||
| 736 | + | } | |||
| 737 | + | } | |||
| 656 | a.setTurnState(TurnIdle, ev.TurnID) | 738 | a.setTurnState(TurnIdle, ev.TurnID) | ||
| 739 | + | notifyIdle = true | |||
| 657 | persist = true | 740 | persist = true | ||
| 658 | case EventAgentExited, "agent_exited", "agent_stopped": | 741 | case EventAgentExited, "agent_exited", "agent_stopped": | ||
| 659 | a.setProcessState(ProcessExited) | 742 | a.setProcessState(ProcessExited) | ||
| 660 | persist = true | 743 | persist = true | ||
| 744 | + | case "error": | |||
| 745 | + | if code, _ := ev.Data["code"].(string); code == "turn_rejected" && eventMatchesPendingResume(a, ev) { | |||
| 746 | + | if a.rejectActiveResumePrompt() { | |||
| 747 | + | reason, _ := ev.Data["reason"].(string) | |||
| 748 | + | if reason == "max_turns" { | |||
| 749 | + | a.setTurnState(TurnFailed, ev.TurnID) | |||
| 750 | + | } else { | |||
| 751 | + | a.setTurnState(TurnIdle, ev.TurnID) | |||
| 752 | + | notifyIdle = true | |||
| 753 | + | } | |||
| 754 | + | persist = true | |||
| 755 | + | } | |||
| 756 | + | } | |||
| 661 | case "turn_end": | 757 | case "turn_end": | ||
| 662 | // Provider/tool-loop turn_end events (for example stop=tool_use) | 758 | // Provider/tool-loop turn_end events (for example stop=tool_use) | ||
| 663 | // do not carry the daemon's prompt step and are not terminal for | 759 | // do not carry the daemon's prompt step and are not terminal for | ||
| 664 | // the delegated task. They remain in the event log, but must not | 760 | // the delegated task. They remain in the event log, but must not | ||
| 665 | // overwrite the delegated turn state or trigger persistence. | 761 | // overwrite the delegated turn state or trigger persistence. | ||
| 666 | - | if _, ok := ev.Data["step"].(float64); !ok { | 762 | + | if !isDelegatedTurnEnd(ev) { |
| 667 | break | 763 | break | ||
| 668 | } | 764 | } | ||
| 669 | if message, _ := ev.Data["error"].(string); message != "" { | 765 | if message, _ := ev.Data["error"].(string); message != "" { | ||
| @@ -674,8 +770,15 @@ func updateAgentFromEvent(a *Agent, ev Event) { | |||||
| 674 | persist = true | 770 | persist = true | ||
| 675 | } | 771 | } | ||
| 676 | if persist && a.persistFn != nil { | 772 | if persist && a.persistFn != nil { | ||
| 677 | - | a.persistFn(a) | 773 | + | if err := a.persistFn(a); err != nil { |
| 774 | + | a.recordPersistenceError(err) | |||
| 775 | + | return err | |||
| 776 | + | } | |||
| 678 | } | 777 | } | ||
| 778 | + | if notifyIdle { | |||
| 779 | + | a.notifyTurnIdle() | |||
| 780 | + | } | |||
| 781 | + | return nil | |||
| 679 | } | 782 | } | ||
| 680 | 783 | ||||
| 681 | // notifyPromptTurnEnd calls Agent.OnTurnEnd only for the subagent | 784 | // notifyPromptTurnEnd calls Agent.OnTurnEnd only for the subagent | ||
| @@ -686,7 +789,10 @@ func notifyPromptTurnEnd(a *Agent, ev Event) { | |||||
| 686 | if a == nil || ev.Type != "turn_end" { | 789 | if a == nil || ev.Type != "turn_end" { | ||
| 687 | return | 790 | return | ||
| 688 | } | 791 | } | ||
| 689 | - | step, ok := ev.Data["step"].(float64) | 792 | + | if !isDelegatedTurnEnd(ev) { |
| 793 | + | return | |||
| 794 | + | } | |||
| 795 | + | step, ok := eventCounter(ev.Data, "step") | |||
| 690 | if !ok { | 796 | if !ok { | ||
| 691 | return | 797 | return | ||
| 692 | } | 798 | } | ||
| @@ -700,7 +806,7 @@ func notifyPromptTurnEnd(a *Agent, ev Event) { | |||||
| 700 | return | 806 | return | ||
| 701 | } | 807 | } | ||
| 702 | a.mu.Unlock() | 808 | a.mu.Unlock() | ||
| 703 | - | go fn(int(step), errMsg) | 809 | + | go fn(step, errMsg) |
| 704 | } | 810 | } | ||
| 705 | 811 | ||||
| 706 | // applyEventToSink translates an Event into Sink updates. Only a | 812 | // applyEventToSink translates an Event into Sink updates. Only a | ||
M packages/agent/subagents/runner_test.go +111 -2
| @@ -383,11 +383,68 @@ func TestDefaultChildArgsResumeUsesFollowUpPrompt(t *testing.T) { | |||
| 383 | 383 | } | |
| 384 | 384 | } | |
| 385 | 385 | ||
| 386 | + | func TestReadyDoesNotMarkInitialDelegatedTurnIdle(t *testing.T) { | |
| 387 | + | a := &Agent{Task: "initial delegated task"} | |
| 388 | + | a.setTurnState(TurnQueued, "") | |
| 389 | + | ||
| 390 | + | if err := updateAgentFromEvent(a, NewEvent(EventAgentReady, map[string]any{"lifetime_turns": 0, "current_run_turns": 0})); err != nil { | |
| 391 | + | t.Fatal(err) | |
| 392 | + | } | |
| 393 | + | if got := a.TurnState(); got != TurnQueued { | |
| 394 | + | t.Fatalf("ready changed turn state to %s, want %s", got, TurnQueued) | |
| 395 | + | } | |
| 396 | + | if err := updateAgentFromEvent(a, NewEvent(EventTurnStarted, map[string]any{ | |
| 397 | + | "turn_id": "turn-1", | |
| 398 | + | "lifetime_turns": 1, | |
| 399 | + | "current_run_turns": 1, | |
| 400 | + | })); err != nil { | |
| 401 | + | t.Fatal(err) | |
| 402 | + | } | |
| 403 | + | if got := a.TurnState(); got != TurnRunning { | |
| 404 | + | t.Fatalf("turn.started changed turn state to %s, want %s", got, TurnRunning) | |
| 405 | + | } | |
| 406 | + | } | |
| 407 | + | ||
| 408 | + | func TestUpdateAgentFromEventIgnoresStaleRejectedFollowUp(t *testing.T) { | |
| 409 | + | a := &Agent{} | |
| 410 | + | a.setResumePrompt("current prompt", time.Now()) | |
| 411 | + | if err := updateAgentFromEvent(a, NewEvent("error", map[string]any{ | |
| 412 | + | "code": "turn_rejected", "reason": "busy", "command_id": "stale-command", | |
| 413 | + | })); err != nil { | |
| 414 | + | t.Fatal(err) | |
| 415 | + | } | |
| 416 | + | if prompt, _ := a.ResumePromptInfo(); prompt != "current prompt" { | |
| 417 | + | t.Fatalf("stale rejection changed active prompt to %q", prompt) | |
| 418 | + | } | |
| 419 | + | } | |
| 420 | + | ||
| 421 | + | func TestUpdateAgentFromEventRecoversRejectedFollowUp(t *testing.T) { | |
| 422 | + | a := &Agent{} | |
| 423 | + | a.setResumePrompt("retry me", time.Now()) | |
| 424 | + | a.setTurnState(TurnQueued, "turn-1") | |
| 425 | + | if err := updateAgentFromEvent(a, NewEvent("error", map[string]any{ | |
| 426 | + | "code": "turn_rejected", | |
| 427 | + | "reason": "busy", | |
| 428 | + | })); err != nil { | |
| 429 | + | t.Fatal(err) | |
| 430 | + | } | |
| 431 | + | if got := a.TurnState(); got != TurnIdle { | |
| 432 | + | t.Fatalf("rejected turn state = %s, want %s", got, TurnIdle) | |
| 433 | + | } | |
| 434 | + | if prompt, _ := a.ResumePromptInfo(); prompt != "" { | |
| 435 | + | t.Fatalf("rejected prompt remained active: %q", prompt) | |
| 436 | + | } | |
| 437 | + | queued := a.resumePromptQueueSnapshot() | |
| 438 | + | if len(queued) != 1 || queued[0].Prompt != "retry me" { | |
| 439 | + | t.Fatalf("rejected prompt queue = %+v, want retry me", queued) | |
| 440 | + | } | |
| 441 | + | } | |
| 442 | + | ||
| 386 | 443 | func TestUpdateAgentFromEventIgnoresProviderToolLoopTurnEnd(t *testing.T) { | |
| 387 | 444 | a := &Agent{} | |
| 388 | 445 | a.setTurnState(TurnRunning, "turn-1") | |
| 389 | 446 | persisted := 0 | |
| 390 | - | a.persistFn = func(*Agent) { persisted++ } | |
| 447 | + | a.persistFn = func(*Agent) error { persisted++; return nil } | |
| 391 | 448 | ||
| 392 | 449 | updateAgentFromEvent(a, NewEvent("turn_end", map[string]any{"stop": "tool_use"})) | |
| 393 | 450 | ||
| @@ -405,7 +462,7 @@ func TestUpdateAgentFromEventIgnoresProviderToolLoopTurnEnd(t *testing.T) { | |||
| 405 | 462 | func TestUpdateAgentFromEventPersistsTerminalTurnEndWithStep(t *testing.T) { | |
| 406 | 463 | a := &Agent{} | |
| 407 | 464 | persisted := 0 | |
| 408 | - | a.persistFn = func(*Agent) { persisted++ } | |
| 465 | + | a.persistFn = func(*Agent) error { persisted++; return nil } | |
| 409 | 466 | ||
| 410 | 467 | updateAgentFromEvent(a, NewEvent("turn_end", map[string]any{"step": float64(2), "error": "boom"})) | |
| 411 | 468 | ||
| @@ -481,6 +538,58 @@ func TestSetOnTurnEndReplaysPendingNoticesInArrivalOrder(t *testing.T) { | |||
| 481 | 538 | } | |
| 482 | 539 | } | |
| 483 | 540 | ||
| 541 | + | func TestUpdateAgentFromEventDoesNotCountProviderTurnStarts(t *testing.T) { | |
| 542 | + | a := &Agent{} | |
| 543 | + | a.setTurnCounts(4, 2) | |
| 544 | + | a.setTurnState(TurnRunning, "turn-1") | |
| 545 | + | persisted := 0 | |
| 546 | + | a.persistFn = func(*Agent) error { persisted++; return nil } | |
| 547 | + | ||
| 548 | + | updateAgentFromEvent(a, NewEvent(EventTurnStarted, map[string]any{"step": float64(7), "nested_turn": true})) | |
| 549 | + | ||
| 550 | + | if got := a.LifetimeTurnsValue(); got != 4 { | |
| 551 | + | t.Fatalf("lifetime turns = %d, want 4", got) | |
| 552 | + | } | |
| 553 | + | if got := a.CurrentRunTurnsValue(); got != 2 { | |
| 554 | + | t.Fatalf("current-run turns = %d, want 2", got) | |
| 555 | + | } | |
| 556 | + | if got := a.TurnState(); got != TurnRunning { | |
| 557 | + | t.Fatalf("turn state = %s, want %s", got, TurnRunning) | |
| 558 | + | } | |
| 559 | + | if persisted != 0 { | |
| 560 | + | t.Fatalf("provider turn start persisted %d times, want 0", persisted) | |
| 561 | + | } | |
| 562 | + | } | |
| 563 | + | ||
| 564 | + | func TestUpdateAgentFromEventPersistsCanonicalTurnCounters(t *testing.T) { | |
| 565 | + | stateDir := t.TempDir() | |
| 566 | + | a := &Agent{ID: "counter-agent", stateDir: stateDir} | |
| 567 | + | a.persistFn = func(agent *Agent) error { | |
| 568 | + | if err := writeAgentMeta(stateDir, agent); err != nil { | |
| 569 | + | t.Errorf("write agent metadata: %v", err) | |
| 570 | + | return err | |
| 571 | + | } | |
| 572 | + | return nil | |
| 573 | + | } | |
| 574 | + | ||
| 575 | + | updateAgentFromEvent(a, NewEvent(EventTurnStarted, map[string]any{ | |
| 576 | + | "turn_id": "turn-3", | |
| 577 | + | "lifetime_turns": 3, | |
| 578 | + | "current_run_turns": 1, | |
| 579 | + | })) | |
| 580 | + | ||
| 581 | + | meta, err := readAgentMeta(stateDir) | |
| 582 | + | if err != nil { | |
| 583 | + | t.Fatal(err) | |
| 584 | + | } | |
| 585 | + | if meta.LifetimeTurns != 3 || meta.CurrentRunTurns != 1 { | |
| 586 | + | t.Fatalf("persisted counters = (%d, %d), want (3, 1)", meta.LifetimeTurns, meta.CurrentRunTurns) | |
| 587 | + | } | |
| 588 | + | if got := a.Snapshot(); got.LifetimeTurns != 3 || got.CurrentRunTurns != 1 { | |
| 589 | + | t.Fatalf("snapshot counters = (%d, %d), want (3, 1)", got.LifetimeTurns, got.CurrentRunTurns) | |
| 590 | + | } | |
| 591 | + | } | |
| 592 | + | ||
| 484 | 593 | func indexOf(xs []string, want string) int { | |
| 485 | 594 | for i, x := range xs { | |
| 486 | 595 | if x == want { | |
| @@ -383,11 +383,68 @@ func TestDefaultChildArgsResumeUsesFollowUpPrompt(t *testing.T) { | |||||
| 383 | } | 383 | } | ||
| 384 | } | 384 | } | ||
| 385 | 385 | ||||
| 386 | + | func TestReadyDoesNotMarkInitialDelegatedTurnIdle(t *testing.T) { | |||
| 387 | + | a := &Agent{Task: "initial delegated task"} | |||
| 388 | + | a.setTurnState(TurnQueued, "") | |||
| 389 | + | ||||
| 390 | + | if err := updateAgentFromEvent(a, NewEvent(EventAgentReady, map[string]any{"lifetime_turns": 0, "current_run_turns": 0})); err != nil { | |||
| 391 | + | t.Fatal(err) | |||
| 392 | + | } | |||
| 393 | + | if got := a.TurnState(); got != TurnQueued { | |||
| 394 | + | t.Fatalf("ready changed turn state to %s, want %s", got, TurnQueued) | |||
| 395 | + | } | |||
| 396 | + | if err := updateAgentFromEvent(a, NewEvent(EventTurnStarted, map[string]any{ | |||
| 397 | + | "turn_id": "turn-1", | |||
| 398 | + | "lifetime_turns": 1, | |||
| 399 | + | "current_run_turns": 1, | |||
| 400 | + | })); err != nil { | |||
| 401 | + | t.Fatal(err) | |||
| 402 | + | } | |||
| 403 | + | if got := a.TurnState(); got != TurnRunning { | |||
| 404 | + | t.Fatalf("turn.started changed turn state to %s, want %s", got, TurnRunning) | |||
| 405 | + | } | |||
| 406 | + | } | |||
| 407 | + | ||||
| 408 | + | func TestUpdateAgentFromEventIgnoresStaleRejectedFollowUp(t *testing.T) { | |||
| 409 | + | a := &Agent{} | |||
| 410 | + | a.setResumePrompt("current prompt", time.Now()) | |||
| 411 | + | if err := updateAgentFromEvent(a, NewEvent("error", map[string]any{ | |||
| 412 | + | "code": "turn_rejected", "reason": "busy", "command_id": "stale-command", | |||
| 413 | + | })); err != nil { | |||
| 414 | + | t.Fatal(err) | |||
| 415 | + | } | |||
| 416 | + | if prompt, _ := a.ResumePromptInfo(); prompt != "current prompt" { | |||
| 417 | + | t.Fatalf("stale rejection changed active prompt to %q", prompt) | |||
| 418 | + | } | |||
| 419 | + | } | |||
| 420 | + | ||||
| 421 | + | func TestUpdateAgentFromEventRecoversRejectedFollowUp(t *testing.T) { | |||
| 422 | + | a := &Agent{} | |||
| 423 | + | a.setResumePrompt("retry me", time.Now()) | |||
| 424 | + | a.setTurnState(TurnQueued, "turn-1") | |||
| 425 | + | if err := updateAgentFromEvent(a, NewEvent("error", map[string]any{ | |||
| 426 | + | "code": "turn_rejected", | |||
| 427 | + | "reason": "busy", | |||
| 428 | + | })); err != nil { | |||
| 429 | + | t.Fatal(err) | |||
| 430 | + | } | |||
| 431 | + | if got := a.TurnState(); got != TurnIdle { | |||
| 432 | + | t.Fatalf("rejected turn state = %s, want %s", got, TurnIdle) | |||
| 433 | + | } | |||
| 434 | + | if prompt, _ := a.ResumePromptInfo(); prompt != "" { | |||
| 435 | + | t.Fatalf("rejected prompt remained active: %q", prompt) | |||
| 436 | + | } | |||
| 437 | + | queued := a.resumePromptQueueSnapshot() | |||
| 438 | + | if len(queued) != 1 || queued[0].Prompt != "retry me" { | |||
| 439 | + | t.Fatalf("rejected prompt queue = %+v, want retry me", queued) | |||
| 440 | + | } | |||
| 441 | + | } | |||
| 442 | + | ||||
| 386 | func TestUpdateAgentFromEventIgnoresProviderToolLoopTurnEnd(t *testing.T) { | 443 | func TestUpdateAgentFromEventIgnoresProviderToolLoopTurnEnd(t *testing.T) { | ||
| 387 | a := &Agent{} | 444 | a := &Agent{} | ||
| 388 | a.setTurnState(TurnRunning, "turn-1") | 445 | a.setTurnState(TurnRunning, "turn-1") | ||
| 389 | persisted := 0 | 446 | persisted := 0 | ||
| 390 | - | a.persistFn = func(*Agent) { persisted++ } | 447 | + | a.persistFn = func(*Agent) error { persisted++; return nil } |
| 391 | 448 | ||||
| 392 | updateAgentFromEvent(a, NewEvent("turn_end", map[string]any{"stop": "tool_use"})) | 449 | updateAgentFromEvent(a, NewEvent("turn_end", map[string]any{"stop": "tool_use"})) | ||
| 393 | 450 | ||||
| @@ -405,7 +462,7 @@ func TestUpdateAgentFromEventIgnoresProviderToolLoopTurnEnd(t *testing.T) { | |||||
| 405 | func TestUpdateAgentFromEventPersistsTerminalTurnEndWithStep(t *testing.T) { | 462 | func TestUpdateAgentFromEventPersistsTerminalTurnEndWithStep(t *testing.T) { | ||
| 406 | a := &Agent{} | 463 | a := &Agent{} | ||
| 407 | persisted := 0 | 464 | persisted := 0 | ||
| 408 | - | a.persistFn = func(*Agent) { persisted++ } | 465 | + | a.persistFn = func(*Agent) error { persisted++; return nil } |
| 409 | 466 | ||||
| 410 | updateAgentFromEvent(a, NewEvent("turn_end", map[string]any{"step": float64(2), "error": "boom"})) | 467 | updateAgentFromEvent(a, NewEvent("turn_end", map[string]any{"step": float64(2), "error": "boom"})) | ||
| 411 | 468 | ||||
| @@ -481,6 +538,58 @@ func TestSetOnTurnEndReplaysPendingNoticesInArrivalOrder(t *testing.T) { | |||||
| 481 | } | 538 | } | ||
| 482 | } | 539 | } | ||
| 483 | 540 | ||||
| 541 | + | func TestUpdateAgentFromEventDoesNotCountProviderTurnStarts(t *testing.T) { | |||
| 542 | + | a := &Agent{} | |||
| 543 | + | a.setTurnCounts(4, 2) | |||
| 544 | + | a.setTurnState(TurnRunning, "turn-1") | |||
| 545 | + | persisted := 0 | |||
| 546 | + | a.persistFn = func(*Agent) error { persisted++; return nil } | |||
| 547 | + | ||||
| 548 | + | updateAgentFromEvent(a, NewEvent(EventTurnStarted, map[string]any{"step": float64(7), "nested_turn": true})) | |||
| 549 | + | ||||
| 550 | + | if got := a.LifetimeTurnsValue(); got != 4 { | |||
| 551 | + | t.Fatalf("lifetime turns = %d, want 4", got) | |||
| 552 | + | } | |||
| 553 | + | if got := a.CurrentRunTurnsValue(); got != 2 { | |||
| 554 | + | t.Fatalf("current-run turns = %d, want 2", got) | |||
| 555 | + | } | |||
| 556 | + | if got := a.TurnState(); got != TurnRunning { | |||
| 557 | + | t.Fatalf("turn state = %s, want %s", got, TurnRunning) | |||
| 558 | + | } | |||
| 559 | + | if persisted != 0 { | |||
| 560 | + | t.Fatalf("provider turn start persisted %d times, want 0", persisted) | |||
| 561 | + | } | |||
| 562 | + | } | |||
| 563 | + | ||||
| 564 | + | func TestUpdateAgentFromEventPersistsCanonicalTurnCounters(t *testing.T) { | |||
| 565 | + | stateDir := t.TempDir() | |||
| 566 | + | a := &Agent{ID: "counter-agent", stateDir: stateDir} | |||
| 567 | + | a.persistFn = func(agent *Agent) error { | |||
| 568 | + | if err := writeAgentMeta(stateDir, agent); err != nil { | |||
| 569 | + | t.Errorf("write agent metadata: %v", err) | |||
| 570 | + | return err | |||
| 571 | + | } | |||
| 572 | + | return nil | |||
| 573 | + | } | |||
| 574 | + | ||||
| 575 | + | updateAgentFromEvent(a, NewEvent(EventTurnStarted, map[string]any{ | |||
| 576 | + | "turn_id": "turn-3", | |||
| 577 | + | "lifetime_turns": 3, | |||
| 578 | + | "current_run_turns": 1, | |||
| 579 | + | })) | |||
| 580 | + | ||||
| 581 | + | meta, err := readAgentMeta(stateDir) | |||
| 582 | + | if err != nil { | |||
| 583 | + | t.Fatal(err) | |||
| 584 | + | } | |||
| 585 | + | if meta.LifetimeTurns != 3 || meta.CurrentRunTurns != 1 { | |||
| 586 | + | t.Fatalf("persisted counters = (%d, %d), want (3, 1)", meta.LifetimeTurns, meta.CurrentRunTurns) | |||
| 587 | + | } | |||
| 588 | + | if got := a.Snapshot(); got.LifetimeTurns != 3 || got.CurrentRunTurns != 1 { | |||
| 589 | + | t.Fatalf("snapshot counters = (%d, %d), want (3, 1)", got.LifetimeTurns, got.CurrentRunTurns) | |||
| 590 | + | } | |||
| 591 | + | } | |||
| 592 | + | ||||
| 484 | func indexOf(xs []string, want string) int { | 593 | func indexOf(xs []string, want string) int { | ||
| 485 | for i, x := range xs { | 594 | for i, x := range xs { | ||
| 486 | if x == want { | 595 | if x == want { | ||
M packages/agent/subagents/scheduler.go +17 -6
| @@ -2,6 +2,7 @@ package subagents | |||
| 2 | 2 | ||
| 3 | 3 | import ( | |
| 4 | 4 | "context" | |
| 5 | + | "errors" | |
| 5 | 6 | "fmt" | |
| 6 | 7 | "os" | |
| 7 | 8 | "path/filepath" | |
| @@ -184,7 +185,13 @@ func (f *Supervisor) schedule() { | |||
| 184 | 185 | a.activity = "starting" | |
| 185 | 186 | a.mu.Unlock() | |
| 186 | 187 | a.setProcessState(ProcessStarting) | |
| 187 | - | a.setTurnState(TurnIdle, "") | |
| 188 | + | prompt, _ := a.ResumePromptInfo() | |
| 189 | + | initialTurn := (!a.Resuming && strings.TrimSpace(a.Task) != "") || prompt != "" | |
| 190 | + | if initialTurn { | |
| 191 | + | a.setTurnState(TurnQueued, "") | |
| 192 | + | } else { | |
| 193 | + | a.setTurnState(TurnIdle, "") | |
| 194 | + | } | |
| 188 | 195 | a.incrementAttempt() | |
| 189 | 196 | ||
| 190 | 197 | f.active++ | |
| @@ -374,13 +381,17 @@ func (f *Supervisor) releaseCapacity(a *Agent) { | |||
| 374 | 381 | f.schedule() | |
| 375 | 382 | } | |
| 376 | 383 | ||
| 377 | - | func (f *Supervisor) persistAgent(a *Agent) { | |
| 384 | + | func (f *Supervisor) persistAgent(a *Agent) error { | |
| 378 | 385 | if a == nil { | |
| 379 | - | return | |
| 386 | + | return errors.New("subagents: nil agent") | |
| 380 | 387 | } | |
| 381 | 388 | stateDir := a.stateDirectory(f.cfg.Root) | |
| 382 | - | if _, err := os.Stat(stateDir); err != nil { | |
| 383 | - | return | |
| 389 | + | if err := writeAgentMeta(stateDir, a); err != nil { | |
| 390 | + | // Lifecycle callbacks cannot all return an error to their caller. Keep | |
| 391 | + | // the failure visible in the agent snapshot instead of silently | |
| 392 | + | // allowing a queued prompt or counter to disappear on restart. | |
| 393 | + | a.recordPersistenceError(err) | |
| 394 | + | return err | |
| 384 | 395 | } | |
| 385 | - | _ = writeAgentMeta(stateDir, a) | |
| 396 | + | return nil | |
| 386 | 397 | } | |
| @@ -2,6 +2,7 @@ package subagents | |||||
| 2 | 2 | ||||
| 3 | import ( | 3 | import ( | ||
| 4 | "context" | 4 | "context" | ||
| 5 | + | "errors" | |||
| 5 | "fmt" | 6 | "fmt" | ||
| 6 | "os" | 7 | "os" | ||
| 7 | "path/filepath" | 8 | "path/filepath" | ||
| @@ -184,7 +185,13 @@ func (f *Supervisor) schedule() { | |||||
| 184 | a.activity = "starting" | 185 | a.activity = "starting" | ||
| 185 | a.mu.Unlock() | 186 | a.mu.Unlock() | ||
| 186 | a.setProcessState(ProcessStarting) | 187 | a.setProcessState(ProcessStarting) | ||
| 187 | - | a.setTurnState(TurnIdle, "") | 188 | + | prompt, _ := a.ResumePromptInfo() |
| 189 | + | initialTurn := (!a.Resuming && strings.TrimSpace(a.Task) != "") || prompt != "" | |||
| 190 | + | if initialTurn { | |||
| 191 | + | a.setTurnState(TurnQueued, "") | |||
| 192 | + | } else { | |||
| 193 | + | a.setTurnState(TurnIdle, "") | |||
| 194 | + | } | |||
| 188 | a.incrementAttempt() | 195 | a.incrementAttempt() | ||
| 189 | 196 | ||||
| 190 | f.active++ | 197 | f.active++ | ||
| @@ -374,13 +381,17 @@ func (f *Supervisor) releaseCapacity(a *Agent) { | |||||
| 374 | f.schedule() | 381 | f.schedule() | ||
| 375 | } | 382 | } | ||
| 376 | 383 | ||||
| 377 | - | func (f *Supervisor) persistAgent(a *Agent) { | 384 | + | func (f *Supervisor) persistAgent(a *Agent) error { |
| 378 | if a == nil { | 385 | if a == nil { | ||
| 379 | - | return | 386 | + | return errors.New("subagents: nil agent") |
| 380 | } | 387 | } | ||
| 381 | stateDir := a.stateDirectory(f.cfg.Root) | 388 | stateDir := a.stateDirectory(f.cfg.Root) | ||
| 382 | - | if _, err := os.Stat(stateDir); err != nil { | 389 | + | if err := writeAgentMeta(stateDir, a); err != nil { |
| 383 | - | return | 390 | + | // Lifecycle callbacks cannot all return an error to their caller. Keep |
| 391 | + | // the failure visible in the agent snapshot instead of silently | |||
| 392 | + | // allowing a queued prompt or counter to disappear on restart. | |||
| 393 | + | a.recordPersistenceError(err) | |||
| 394 | + | return err | |||
| 384 | } | 395 | } | ||
| 385 | - | _ = writeAgentMeta(stateDir, a) | 396 | + | return nil |
| 386 | } | 397 | } | ||
M packages/agent/subagents/supervisor.go +6 -1
| @@ -543,6 +543,7 @@ func (f *Supervisor) SpawnReq(ctx context.Context, req SpawnRequest) (*Agent, er | |||
| 543 | 543 | } | |
| 544 | 544 | a.ctx, a.cancel = runCtx, cancel | |
| 545 | 545 | a.persistFn = f.persistAgent | |
| 546 | + | a.setOnTurnIdle(func() { f.dispatchQueuedResume(a) }) | |
| 546 | 547 | a.workspaceCleanup = func() error { return workspace.Cleanup(context.Background()) } | |
| 547 | 548 | a.workspaceCapture = func() (WorkspaceCapture, error) { return workspace.Capture(context.Background()) } | |
| 548 | 549 | a.runner = f.cfg.NewRunner(a) | |
| @@ -1075,6 +1076,8 @@ type AgentSnapshot struct { | |||
| 1075 | 1076 | ProcessState ProcessState | |
| 1076 | 1077 | TurnState TurnState | |
| 1077 | 1078 | CurrentTurnID string | |
| 1079 | + | LifetimeTurns int | |
| 1080 | + | CurrentRunTurns int | |
| 1078 | 1081 | Attempt int | |
| 1079 | 1082 | Activity string | |
| 1080 | 1083 | Started time.Time | |
| @@ -1136,6 +1139,8 @@ func (a *Agent) Snapshot() AgentSnapshot { | |||
| 1136 | 1139 | processState := a.processState | |
| 1137 | 1140 | turnState := a.turnState | |
| 1138 | 1141 | currentTurnID := a.currentTurnID | |
| 1142 | + | lifetimeTurns := a.LifetimeTurns | |
| 1143 | + | currentRunTurns := a.CurrentRunTurns | |
| 1139 | 1144 | attempt := a.Attempt | |
| 1140 | 1145 | lastActivity := a.lastActivity | |
| 1141 | 1146 | updatedAt := a.updatedAt | |
| @@ -1148,7 +1153,7 @@ func (a *Agent) Snapshot() AgentSnapshot { | |||
| 1148 | 1153 | return AgentSnapshot{ | |
| 1149 | 1154 | ID: a.ID, Task: a.Task, Dir: a.Dir, | |
| 1150 | 1155 | Status: status, ProcessState: processState, TurnState: turnState, | |
| 1151 | - | CurrentTurnID: currentTurnID, Attempt: attempt, Activity: activity, | |
| 1156 | + | CurrentTurnID: currentTurnID, LifetimeTurns: lifetimeTurns, CurrentRunTurns: currentRunTurns, Attempt: attempt, Activity: activity, | |
| 1152 | 1157 | Started: a.Started, Finished: finished, CreatedAt: a.Started, UpdatedAt: updatedAt, LastActivity: lastActivity, | |
| 1153 | 1158 | Err: errStr, Tail: tail, Lines: lines, | |
| 1154 | 1159 | LastAssistant: lastAssistant, Result: result, ResultRef: resultRef, | |
| @@ -543,6 +543,7 @@ func (f *Supervisor) SpawnReq(ctx context.Context, req SpawnRequest) (*Agent, er | |||||
| 543 | } | 543 | } | ||
| 544 | a.ctx, a.cancel = runCtx, cancel | 544 | a.ctx, a.cancel = runCtx, cancel | ||
| 545 | a.persistFn = f.persistAgent | 545 | a.persistFn = f.persistAgent | ||
| 546 | + | a.setOnTurnIdle(func() { f.dispatchQueuedResume(a) }) | |||
| 546 | a.workspaceCleanup = func() error { return workspace.Cleanup(context.Background()) } | 547 | a.workspaceCleanup = func() error { return workspace.Cleanup(context.Background()) } | ||
| 547 | a.workspaceCapture = func() (WorkspaceCapture, error) { return workspace.Capture(context.Background()) } | 548 | a.workspaceCapture = func() (WorkspaceCapture, error) { return workspace.Capture(context.Background()) } | ||
| 548 | a.runner = f.cfg.NewRunner(a) | 549 | a.runner = f.cfg.NewRunner(a) | ||
| @@ -1075,6 +1076,8 @@ type AgentSnapshot struct { | |||||
| 1075 | ProcessState ProcessState | 1076 | ProcessState ProcessState | ||
| 1076 | TurnState TurnState | 1077 | TurnState TurnState | ||
| 1077 | CurrentTurnID string | 1078 | CurrentTurnID string | ||
| 1079 | + | LifetimeTurns int | |||
| 1080 | + | CurrentRunTurns int | |||
| 1078 | Attempt int | 1081 | Attempt int | ||
| 1079 | Activity string | 1082 | Activity string | ||
| 1080 | Started time.Time | 1083 | Started time.Time | ||
| @@ -1136,6 +1139,8 @@ func (a *Agent) Snapshot() AgentSnapshot { | |||||
| 1136 | processState := a.processState | 1139 | processState := a.processState | ||
| 1137 | turnState := a.turnState | 1140 | turnState := a.turnState | ||
| 1138 | currentTurnID := a.currentTurnID | 1141 | currentTurnID := a.currentTurnID | ||
| 1142 | + | lifetimeTurns := a.LifetimeTurns | |||
| 1143 | + | currentRunTurns := a.CurrentRunTurns | |||
| 1139 | attempt := a.Attempt | 1144 | attempt := a.Attempt | ||
| 1140 | lastActivity := a.lastActivity | 1145 | lastActivity := a.lastActivity | ||
| 1141 | updatedAt := a.updatedAt | 1146 | updatedAt := a.updatedAt | ||
| @@ -1148,7 +1153,7 @@ func (a *Agent) Snapshot() AgentSnapshot { | |||||
| 1148 | return AgentSnapshot{ | 1153 | return AgentSnapshot{ | ||
| 1149 | ID: a.ID, Task: a.Task, Dir: a.Dir, | 1154 | ID: a.ID, Task: a.Task, Dir: a.Dir, | ||
| 1150 | Status: status, ProcessState: processState, TurnState: turnState, | 1155 | Status: status, ProcessState: processState, TurnState: turnState, | ||
| 1151 | - | CurrentTurnID: currentTurnID, Attempt: attempt, Activity: activity, | 1156 | + | CurrentTurnID: currentTurnID, LifetimeTurns: lifetimeTurns, CurrentRunTurns: currentRunTurns, Attempt: attempt, Activity: activity, |
| 1152 | Started: a.Started, Finished: finished, CreatedAt: a.Started, UpdatedAt: updatedAt, LastActivity: lastActivity, | 1157 | Started: a.Started, Finished: finished, CreatedAt: a.Started, UpdatedAt: updatedAt, LastActivity: lastActivity, | ||
| 1153 | Err: errStr, Tail: tail, Lines: lines, | 1158 | Err: errStr, Tail: tail, Lines: lines, | ||
| 1154 | LastAssistant: lastAssistant, Result: result, ResultRef: resultRef, | 1159 | LastAssistant: lastAssistant, Result: result, ResultRef: resultRef, | ||
A packages/agent/subagents/turn_classification.go +98 -0
| @@ -0,0 +1,98 @@ | |||
| 1 | + | package subagents | |
| 2 | + | ||
| 3 | + | import "fmt" | |
| 4 | + | ||
| 5 | + | // turnEventClass identifies the boundary that owns a turn event. The worker | |
| 6 | + | // daemon's turn.started event is the delegated message boundary; provider | |
| 7 | + | // turn_start events are nested model-loop activity. New workers carry stable | |
| 8 | + | // lifetime/current counters on the delegated boundary and nested_turn on the | |
| 9 | + | // provider boundary. The fallbacks preserve protocol-v1 logs, which used | |
| 10 | + | // turn.started without counters for delegated turns and turn_start for nested | |
| 11 | + | // turns. | |
| 12 | + | type turnEventClass uint8 | |
| 13 | + | ||
| 14 | + | const ( | |
| 15 | + | turnEventUnknown turnEventClass = iota | |
| 16 | + | turnEventDelegated | |
| 17 | + | turnEventNested | |
| 18 | + | ) | |
| 19 | + | ||
| 20 | + | func classifyTurnEvent(ev Event) turnEventClass { | |
| 21 | + | if nested, ok := ev.Data["nested_turn"].(bool); ok && nested { | |
| 22 | + | return turnEventNested | |
| 23 | + | } | |
| 24 | + | switch ev.Type { | |
| 25 | + | case "turn_start": | |
| 26 | + | return turnEventNested | |
| 27 | + | case EventTurnStarted: | |
| 28 | + | if _, lifetimeOK := eventCounter(ev.Data, "lifetime_turns"); lifetimeOK { | |
| 29 | + | if _, currentOK := eventCounter(ev.Data, "current_run_turns"); currentOK { | |
| 30 | + | return turnEventDelegated | |
| 31 | + | } | |
| 32 | + | } | |
| 33 | + | // v1 turn.started without counters is the delegated boundary. A | |
| 34 | + | // current worker marks nested canonical events explicitly above. | |
| 35 | + | return turnEventDelegated | |
| 36 | + | case "turn_end": | |
| 37 | + | if _, ok := eventCounter(ev.Data, "step"); ok { | |
| 38 | + | return turnEventDelegated | |
| 39 | + | } | |
| 40 | + | return turnEventNested | |
| 41 | + | } | |
| 42 | + | return turnEventUnknown | |
| 43 | + | } | |
| 44 | + | ||
| 45 | + | func isDelegatedTurnStart(ev Event) bool { | |
| 46 | + | return ev.Type == EventTurnStarted && classifyTurnEvent(ev) == turnEventDelegated | |
| 47 | + | } | |
| 48 | + | ||
| 49 | + | // IsDelegatedTurnStart is shared with the worker's replay bootstrap so live, | |
| 50 | + | // replayed, and protocol-v1 event handling use the same boundary rules. | |
| 51 | + | func IsDelegatedTurnStart(ev Event) bool { | |
| 52 | + | return isDelegatedTurnStart(ev) | |
| 53 | + | } | |
| 54 | + | ||
| 55 | + | func isDelegatedTurnEnd(ev Event) bool { | |
| 56 | + | return ev.Type == "turn_end" && classifyTurnEvent(ev) == turnEventDelegated | |
| 57 | + | } | |
| 58 | + | ||
| 59 | + | func eventTurnID(ev Event) string { | |
| 60 | + | if ev.TurnID != "" { | |
| 61 | + | return ev.TurnID | |
| 62 | + | } | |
| 63 | + | if step, ok := eventCounter(ev.Data, "step"); ok { | |
| 64 | + | return fmt.Sprintf("turn-%d", step) | |
| 65 | + | } | |
| 66 | + | return "" | |
| 67 | + | } | |
| 68 | + | ||
| 69 | + | func userMessageText(ev Event) string { | |
| 70 | + | if ev.Type != "user_message" { | |
| 71 | + | return "" | |
| 72 | + | } | |
| 73 | + | content := ev.Data["content"] | |
| 74 | + | var text string | |
| 75 | + | appendBlock := func(fields map[string]any) { | |
| 76 | + | if typ, _ := fields["type"].(string); typ == "text" { | |
| 77 | + | if value, _ := fields["text"].(string); value != "" { | |
| 78 | + | if text != "" { | |
| 79 | + | text += "\n" | |
| 80 | + | } | |
| 81 | + | text += value | |
| 82 | + | } | |
| 83 | + | } | |
| 84 | + | } | |
| 85 | + | switch blocks := content.(type) { | |
| 86 | + | case []any: | |
| 87 | + | for _, block := range blocks { | |
| 88 | + | if fields, ok := block.(map[string]any); ok { | |
| 89 | + | appendBlock(fields) | |
| 90 | + | } | |
| 91 | + | } | |
| 92 | + | case []map[string]any: | |
| 93 | + | for _, fields := range blocks { | |
| 94 | + | appendBlock(fields) | |
| 95 | + | } | |
| 96 | + | } | |
| 97 | + | return text | |
| 98 | + | } | |
| @@ -0,0 +1,98 @@ | |||||
| 1 | + | package subagents | |||
| 2 | + | ||||
| 3 | + | import "fmt" | |||
| 4 | + | ||||
| 5 | + | // turnEventClass identifies the boundary that owns a turn event. The worker | |||
| 6 | + | // daemon's turn.started event is the delegated message boundary; provider | |||
| 7 | + | // turn_start events are nested model-loop activity. New workers carry stable | |||
| 8 | + | // lifetime/current counters on the delegated boundary and nested_turn on the | |||
| 9 | + | // provider boundary. The fallbacks preserve protocol-v1 logs, which used | |||
| 10 | + | // turn.started without counters for delegated turns and turn_start for nested | |||
| 11 | + | // turns. | |||
| 12 | + | type turnEventClass uint8 | |||
| 13 | + | ||||
| 14 | + | const ( | |||
| 15 | + | turnEventUnknown turnEventClass = iota | |||
| 16 | + | turnEventDelegated | |||
| 17 | + | turnEventNested | |||
| 18 | + | ) | |||
| 19 | + | ||||
| 20 | + | func classifyTurnEvent(ev Event) turnEventClass { | |||
| 21 | + | if nested, ok := ev.Data["nested_turn"].(bool); ok && nested { | |||
| 22 | + | return turnEventNested | |||
| 23 | + | } | |||
| 24 | + | switch ev.Type { | |||
| 25 | + | case "turn_start": | |||
| 26 | + | return turnEventNested | |||
| 27 | + | case EventTurnStarted: | |||
| 28 | + | if _, lifetimeOK := eventCounter(ev.Data, "lifetime_turns"); lifetimeOK { | |||
| 29 | + | if _, currentOK := eventCounter(ev.Data, "current_run_turns"); currentOK { | |||
| 30 | + | return turnEventDelegated | |||
| 31 | + | } | |||
| 32 | + | } | |||
| 33 | + | // v1 turn.started without counters is the delegated boundary. A | |||
| 34 | + | // current worker marks nested canonical events explicitly above. | |||
| 35 | + | return turnEventDelegated | |||
| 36 | + | case "turn_end": | |||
| 37 | + | if _, ok := eventCounter(ev.Data, "step"); ok { | |||
| 38 | + | return turnEventDelegated | |||
| 39 | + | } | |||
| 40 | + | return turnEventNested | |||
| 41 | + | } | |||
| 42 | + | return turnEventUnknown | |||
| 43 | + | } | |||
| 44 | + | ||||
| 45 | + | func isDelegatedTurnStart(ev Event) bool { | |||
| 46 | + | return ev.Type == EventTurnStarted && classifyTurnEvent(ev) == turnEventDelegated | |||
| 47 | + | } | |||
| 48 | + | ||||
| 49 | + | // IsDelegatedTurnStart is shared with the worker's replay bootstrap so live, | |||
| 50 | + | // replayed, and protocol-v1 event handling use the same boundary rules. | |||
| 51 | + | func IsDelegatedTurnStart(ev Event) bool { | |||
| 52 | + | return isDelegatedTurnStart(ev) | |||
| 53 | + | } | |||
| 54 | + | ||||
| 55 | + | func isDelegatedTurnEnd(ev Event) bool { | |||
| 56 | + | return ev.Type == "turn_end" && classifyTurnEvent(ev) == turnEventDelegated | |||
| 57 | + | } | |||
| 58 | + | ||||
| 59 | + | func eventTurnID(ev Event) string { | |||
| 60 | + | if ev.TurnID != "" { | |||
| 61 | + | return ev.TurnID | |||
| 62 | + | } | |||
| 63 | + | if step, ok := eventCounter(ev.Data, "step"); ok { | |||
| 64 | + | return fmt.Sprintf("turn-%d", step) | |||
| 65 | + | } | |||
| 66 | + | return "" | |||
| 67 | + | } | |||
| 68 | + | ||||
| 69 | + | func userMessageText(ev Event) string { | |||
| 70 | + | if ev.Type != "user_message" { | |||
| 71 | + | return "" | |||
| 72 | + | } | |||
| 73 | + | content := ev.Data["content"] | |||
| 74 | + | var text string | |||
| 75 | + | appendBlock := func(fields map[string]any) { | |||
| 76 | + | if typ, _ := fields["type"].(string); typ == "text" { | |||
| 77 | + | if value, _ := fields["text"].(string); value != "" { | |||
| 78 | + | if text != "" { | |||
| 79 | + | text += "\n" | |||
| 80 | + | } | |||
| 81 | + | text += value | |||
| 82 | + | } | |||
| 83 | + | } | |||
| 84 | + | } | |||
| 85 | + | switch blocks := content.(type) { | |||
| 86 | + | case []any: | |||
| 87 | + | for _, block := range blocks { | |||
| 88 | + | if fields, ok := block.(map[string]any); ok { | |||
| 89 | + | appendBlock(fields) | |||
| 90 | + | } | |||
| 91 | + | } | |||
| 92 | + | case []map[string]any: | |||
| 93 | + | for _, fields := range blocks { | |||
| 94 | + | appendBlock(fields) | |||
| 95 | + | } | |||
| 96 | + | } | |||
| 97 | + | return text | |||
| 98 | + | } | |||
A packages/agent/subagents/turn_classification_test.go +63 -0
| @@ -0,0 +1,63 @@ | |||
| 1 | + | package subagents | |
| 2 | + | ||
| 3 | + | import ( | |
| 4 | + | "testing" | |
| 5 | + | "time" | |
| 6 | + | ) | |
| 7 | + | ||
| 8 | + | func TestTurnClassificationUsesStableBoundaryFieldsAcrossProtocolVersions(t *testing.T) { | |
| 9 | + | cases := []struct { | |
| 10 | + | name string | |
| 11 | + | event Event | |
| 12 | + | class turnEventClass | |
| 13 | + | delegated bool | |
| 14 | + | }{ | |
| 15 | + | { | |
| 16 | + | name: "current delegated", | |
| 17 | + | event: NewEvent(EventTurnStarted, map[string]any{ | |
| 18 | + | "lifetime_turns": 4, | |
| 19 | + | "current_run_turns": 2, | |
| 20 | + | }), | |
| 21 | + | class: turnEventDelegated, | |
| 22 | + | delegated: true, | |
| 23 | + | }, | |
| 24 | + | { | |
| 25 | + | name: "current nested", | |
| 26 | + | event: NewEvent(EventTurnStarted, map[string]any{ | |
| 27 | + | "nested_turn": true, | |
| 28 | + | "step": 3, | |
| 29 | + | }), | |
| 30 | + | class: turnEventNested, | |
| 31 | + | }, | |
| 32 | + | { | |
| 33 | + | name: "protocol v1 delegated fallback", | |
| 34 | + | event: Event{Type: EventTurnStarted, Version: 1, Time: time.Now()}, | |
| 35 | + | class: turnEventDelegated, | |
| 36 | + | delegated: true, | |
| 37 | + | }, | |
| 38 | + | { | |
| 39 | + | name: "protocol v1 nested turn_start", | |
| 40 | + | event: Event{Type: "turn_start", Version: 1, Time: time.Now()}, | |
| 41 | + | class: turnEventNested, | |
| 42 | + | }, | |
| 43 | + | } | |
| 44 | + | for _, tc := range cases { | |
| 45 | + | t.Run(tc.name, func(t *testing.T) { | |
| 46 | + | if got := classifyTurnEvent(tc.event); got != tc.class { | |
| 47 | + | t.Fatalf("classifyTurnEvent = %d, want %d", got, tc.class) | |
| 48 | + | } | |
| 49 | + | if got := isDelegatedTurnStart(tc.event); got != tc.delegated { | |
| 50 | + | t.Fatalf("isDelegatedTurnStart = %v, want %v", got, tc.delegated) | |
| 51 | + | } | |
| 52 | + | }) | |
| 53 | + | } | |
| 54 | + | } | |
| 55 | + | ||
| 56 | + | func TestTurnClassificationKeepsNestedAndDelegatedEndsSeparate(t *testing.T) { | |
| 57 | + | if isDelegatedTurnEnd(NewEvent("turn_end", map[string]any{"stop": "tool_use"})) { | |
| 58 | + | t.Fatal("provider turn_end was classified as delegated") | |
| 59 | + | } | |
| 60 | + | if !isDelegatedTurnEnd(NewEvent("turn_end", map[string]any{"step": 2})) { | |
| 61 | + | t.Fatal("worker turn_end was not classified as delegated") | |
| 62 | + | } | |
| 63 | + | } | |
| @@ -0,0 +1,63 @@ | |||||
| 1 | + | package subagents | |||
| 2 | + | ||||
| 3 | + | import ( | |||
| 4 | + | "testing" | |||
| 5 | + | "time" | |||
| 6 | + | ) | |||
| 7 | + | ||||
| 8 | + | func TestTurnClassificationUsesStableBoundaryFieldsAcrossProtocolVersions(t *testing.T) { | |||
| 9 | + | cases := []struct { | |||
| 10 | + | name string | |||
| 11 | + | event Event | |||
| 12 | + | class turnEventClass | |||
| 13 | + | delegated bool | |||
| 14 | + | }{ | |||
| 15 | + | { | |||
| 16 | + | name: "current delegated", | |||
| 17 | + | event: NewEvent(EventTurnStarted, map[string]any{ | |||
| 18 | + | "lifetime_turns": 4, | |||
| 19 | + | "current_run_turns": 2, | |||
| 20 | + | }), | |||
| 21 | + | class: turnEventDelegated, | |||
| 22 | + | delegated: true, | |||
| 23 | + | }, | |||
| 24 | + | { | |||
| 25 | + | name: "current nested", | |||
| 26 | + | event: NewEvent(EventTurnStarted, map[string]any{ | |||
| 27 | + | "nested_turn": true, | |||
| 28 | + | "step": 3, | |||
| 29 | + | }), | |||
| 30 | + | class: turnEventNested, | |||
| 31 | + | }, | |||
| 32 | + | { | |||
| 33 | + | name: "protocol v1 delegated fallback", | |||
| 34 | + | event: Event{Type: EventTurnStarted, Version: 1, Time: time.Now()}, | |||
| 35 | + | class: turnEventDelegated, | |||
| 36 | + | delegated: true, | |||
| 37 | + | }, | |||
| 38 | + | { | |||
| 39 | + | name: "protocol v1 nested turn_start", | |||
| 40 | + | event: Event{Type: "turn_start", Version: 1, Time: time.Now()}, | |||
| 41 | + | class: turnEventNested, | |||
| 42 | + | }, | |||
| 43 | + | } | |||
| 44 | + | for _, tc := range cases { | |||
| 45 | + | t.Run(tc.name, func(t *testing.T) { | |||
| 46 | + | if got := classifyTurnEvent(tc.event); got != tc.class { | |||
| 47 | + | t.Fatalf("classifyTurnEvent = %d, want %d", got, tc.class) | |||
| 48 | + | } | |||
| 49 | + | if got := isDelegatedTurnStart(tc.event); got != tc.delegated { | |||
| 50 | + | t.Fatalf("isDelegatedTurnStart = %v, want %v", got, tc.delegated) | |||
| 51 | + | } | |||
| 52 | + | }) | |||
| 53 | + | } | |||
| 54 | + | } | |||
| 55 | + | ||||
| 56 | + | func TestTurnClassificationKeepsNestedAndDelegatedEndsSeparate(t *testing.T) { | |||
| 57 | + | if isDelegatedTurnEnd(NewEvent("turn_end", map[string]any{"stop": "tool_use"})) { | |||
| 58 | + | t.Fatal("provider turn_end was classified as delegated") | |||
| 59 | + | } | |||
| 60 | + | if !isDelegatedTurnEnd(NewEvent("turn_end", map[string]any{"step": 2})) { | |||
| 61 | + | t.Fatal("worker turn_end was not classified as delegated") | |||
| 62 | + | } | |||
| 63 | + | } | |||
M packages/agent/tools/subagent_resume.go +7 -5
| @@ -10,10 +10,12 @@ import ( | |||
| 10 | 10 | "github.com/bnema/zut/packages/core" | |
| 11 | 11 | ) | |
| 12 | 12 | ||
| 13 | - | // SubagentResumeTool gives a completed or stopped sub-agent a follow-up turn | |
| 14 | - | // while preserving its existing session context. For an idle live worker, the | |
| 15 | - | // follow-up is delivered directly. For a terminal worker, the supervisor | |
| 16 | - | // restarts the retained session with the follow-up as its initial turn. | |
| 13 | + | // SubagentResumeTool gives a sub-agent a follow-up turn while preserving its | |
| 14 | + | // existing session context. Every explicit follow-up starts a fresh max_turns | |
| 15 | + | // budget. For an idle live worker, the follow-up is delivered directly; for an | |
| 16 | + | // active live worker, it is queued for the next available message turn; for a | |
| 17 | + | // terminal worker, the supervisor restarts the retained session with the | |
| 18 | + | // follow-up as its initial turn. | |
| 17 | 19 | type SubagentResumeTool struct { | |
| 18 | 20 | Supervisor *subagents.Supervisor | |
| 19 | 21 | Enabled func() bool | |
| @@ -48,7 +50,7 @@ const subagentResumeSchema = `{ | |||
| 48 | 50 | func (t *SubagentResumeTool) Name() string { return "subagent_resume" } | |
| 49 | 51 | ||
| 50 | 52 | func (t *SubagentResumeTool) Description() string { | |
| 51 | - | return "Send a follow-up to an idle sub-agent, or restart a stopped one with its prior session context preserved." | |
| 53 | + | return "Start a fresh-budget follow-up for a sub-agent, queue it if the worker is active, or restart a stopped one with its prior session context preserved." | |
| 52 | 54 | } | |
| 53 | 55 | ||
| 54 | 56 | func (t *SubagentResumeTool) Schema() json.RawMessage { | |
| @@ -10,10 +10,12 @@ import ( | |||||
| 10 | "github.com/bnema/zut/packages/core" | 10 | "github.com/bnema/zut/packages/core" | ||
| 11 | ) | 11 | ) | ||
| 12 | 12 | ||||
| 13 | - | // SubagentResumeTool gives a completed or stopped sub-agent a follow-up turn | 13 | + | // SubagentResumeTool gives a sub-agent a follow-up turn while preserving its |
| 14 | - | // while preserving its existing session context. For an idle live worker, the | 14 | + | // existing session context. Every explicit follow-up starts a fresh max_turns |
| 15 | - | // follow-up is delivered directly. For a terminal worker, the supervisor | 15 | + | // budget. For an idle live worker, the follow-up is delivered directly; for an |
| 16 | - | // restarts the retained session with the follow-up as its initial turn. | 16 | + | // active live worker, it is queued for the next available message turn; for a |
| 17 | + | // terminal worker, the supervisor restarts the retained session with the | |||
| 18 | + | // follow-up as its initial turn. | |||
| 17 | type SubagentResumeTool struct { | 19 | type SubagentResumeTool struct { | ||
| 18 | Supervisor *subagents.Supervisor | 20 | Supervisor *subagents.Supervisor | ||
| 19 | Enabled func() bool | 21 | Enabled func() bool | ||
| @@ -48,7 +50,7 @@ const subagentResumeSchema = `{ | |||||
| 48 | func (t *SubagentResumeTool) Name() string { return "subagent_resume" } | 50 | func (t *SubagentResumeTool) Name() string { return "subagent_resume" } | ||
| 49 | 51 | ||||
| 50 | func (t *SubagentResumeTool) Description() string { | 52 | func (t *SubagentResumeTool) Description() string { | ||
| 51 | - | return "Send a follow-up to an idle sub-agent, or restart a stopped one with its prior session context preserved." | 53 | + | return "Start a fresh-budget follow-up for a sub-agent, queue it if the worker is active, or restart a stopped one with its prior session context preserved." |
| 52 | } | 54 | } | ||
| 53 | 55 | ||||
| 54 | func (t *SubagentResumeTool) Schema() json.RawMessage { | 56 | func (t *SubagentResumeTool) Schema() json.RawMessage { | ||
M packages/agent/tools/subagent_status.go +20 -16
| @@ -34,15 +34,17 @@ type subagentStatusResponse struct { | |||
| 34 | 34 | } | |
| 35 | 35 | ||
| 36 | 36 | type subagentStatusEntry struct { | |
| 37 | - | ID string `json:"agent_id"` | |
| 38 | - | State string `json:"state"` | |
| 39 | - | ProcessState string `json:"process_state"` | |
| 40 | - | TurnState string `json:"turn_state"` | |
| 41 | - | StartedAt time.Time `json:"started_at"` | |
| 42 | - | UpdatedAt time.Time `json:"updated_at"` | |
| 43 | - | FinishedAt *time.Time `json:"finished_at,omitempty"` | |
| 44 | - | TaskSummary string `json:"task_summary,omitempty"` | |
| 45 | - | Result *subagentStatusResult `json:"result,omitempty"` | |
| 37 | + | ID string `json:"agent_id"` | |
| 38 | + | State string `json:"state"` | |
| 39 | + | ProcessState string `json:"process_state"` | |
| 40 | + | TurnState string `json:"turn_state"` | |
| 41 | + | LifetimeTurns int `json:"lifetime_turns"` | |
| 42 | + | CurrentRunTurns int `json:"current_run_turns"` | |
| 43 | + | StartedAt time.Time `json:"started_at"` | |
| 44 | + | UpdatedAt time.Time `json:"updated_at"` | |
| 45 | + | FinishedAt *time.Time `json:"finished_at,omitempty"` | |
| 46 | + | TaskSummary string `json:"task_summary,omitempty"` | |
| 47 | + | Result *subagentStatusResult `json:"result,omitempty"` | |
| 46 | 48 | } | |
| 47 | 49 | ||
| 48 | 50 | type subagentStatusResult struct { | |
| @@ -132,13 +134,15 @@ func findSubagentStatusSnapshot(snapshots []subagents.AgentSnapshot, id string) | |||
| 132 | 134 | ||
| 133 | 135 | func publicSubagentStatus(snapshot subagents.AgentSnapshot) subagentStatusEntry { | |
| 134 | 136 | entry := subagentStatusEntry{ | |
| 135 | - | ID: snapshot.ID, | |
| 136 | - | State: publicSubagentState(snapshot), | |
| 137 | - | ProcessState: string(snapshot.ProcessState), | |
| 138 | - | TurnState: string(snapshot.TurnState), | |
| 139 | - | StartedAt: snapshot.Started, | |
| 140 | - | UpdatedAt: snapshot.UpdatedAt, | |
| 141 | - | TaskSummary: summarizeSubagentTask(snapshot.Task), | |
| 137 | + | ID: snapshot.ID, | |
| 138 | + | State: publicSubagentState(snapshot), | |
| 139 | + | ProcessState: string(snapshot.ProcessState), | |
| 140 | + | TurnState: string(snapshot.TurnState), | |
| 141 | + | LifetimeTurns: snapshot.LifetimeTurns, | |
| 142 | + | CurrentRunTurns: snapshot.CurrentRunTurns, | |
| 143 | + | StartedAt: snapshot.Started, | |
| 144 | + | UpdatedAt: snapshot.UpdatedAt, | |
| 145 | + | TaskSummary: summarizeSubagentTask(snapshot.Task), | |
| 142 | 146 | } | |
| 143 | 147 | if !snapshot.Finished.IsZero() { | |
| 144 | 148 | finished := snapshot.Finished | |
| @@ -34,15 +34,17 @@ type subagentStatusResponse struct { | |||||
| 34 | } | 34 | } | ||
| 35 | 35 | ||||
| 36 | type subagentStatusEntry struct { | 36 | type subagentStatusEntry struct { | ||
| 37 | - | ID string `json:"agent_id"` | 37 | + | ID string `json:"agent_id"` |
| 38 | - | State string `json:"state"` | 38 | + | State string `json:"state"` |
| 39 | - | ProcessState string `json:"process_state"` | 39 | + | ProcessState string `json:"process_state"` |
| 40 | - | TurnState string `json:"turn_state"` | 40 | + | TurnState string `json:"turn_state"` |
| 41 | - | StartedAt time.Time `json:"started_at"` | 41 | + | LifetimeTurns int `json:"lifetime_turns"` |
| 42 | - | UpdatedAt time.Time `json:"updated_at"` | 42 | + | CurrentRunTurns int `json:"current_run_turns"` |
| 43 | - | FinishedAt *time.Time `json:"finished_at,omitempty"` | 43 | + | StartedAt time.Time `json:"started_at"` |
| 44 | - | TaskSummary string `json:"task_summary,omitempty"` | 44 | + | UpdatedAt time.Time `json:"updated_at"` |
| 45 | - | Result *subagentStatusResult `json:"result,omitempty"` | 45 | + | FinishedAt *time.Time `json:"finished_at,omitempty"` |
| 46 | + | TaskSummary string `json:"task_summary,omitempty"` | |||
| 47 | + | Result *subagentStatusResult `json:"result,omitempty"` | |||
| 46 | } | 48 | } | ||
| 47 | 49 | ||||
| 48 | type subagentStatusResult struct { | 50 | type subagentStatusResult struct { | ||
| @@ -132,13 +134,15 @@ func findSubagentStatusSnapshot(snapshots []subagents.AgentSnapshot, id string) | |||||
| 132 | 134 | ||||
| 133 | func publicSubagentStatus(snapshot subagents.AgentSnapshot) subagentStatusEntry { | 135 | func publicSubagentStatus(snapshot subagents.AgentSnapshot) subagentStatusEntry { | ||
| 134 | entry := subagentStatusEntry{ | 136 | entry := subagentStatusEntry{ | ||
| 135 | - | ID: snapshot.ID, | 137 | + | ID: snapshot.ID, |
| 136 | - | State: publicSubagentState(snapshot), | 138 | + | State: publicSubagentState(snapshot), |
| 137 | - | ProcessState: string(snapshot.ProcessState), | 139 | + | ProcessState: string(snapshot.ProcessState), |
| 138 | - | TurnState: string(snapshot.TurnState), | 140 | + | TurnState: string(snapshot.TurnState), |
| 139 | - | StartedAt: snapshot.Started, | 141 | + | LifetimeTurns: snapshot.LifetimeTurns, |
| 140 | - | UpdatedAt: snapshot.UpdatedAt, | 142 | + | CurrentRunTurns: snapshot.CurrentRunTurns, |
| 141 | - | TaskSummary: summarizeSubagentTask(snapshot.Task), | 143 | + | StartedAt: snapshot.Started, |
| 144 | + | UpdatedAt: snapshot.UpdatedAt, | |||
| 145 | + | TaskSummary: summarizeSubagentTask(snapshot.Task), | |||
| 142 | } | 146 | } | ||
| 143 | if !snapshot.Finished.IsZero() { | 147 | if !snapshot.Finished.IsZero() { | ||
| 144 | finished := snapshot.Finished | 148 | finished := snapshot.Finished | ||
M packages/agent/tools/subagent_status_test.go +22 -0
| @@ -39,6 +39,28 @@ func (r statusTestRunner) Run(ctx context.Context, sink subagents.Sink) error { | |||
| 39 | 39 | } | |
| 40 | 40 | } | |
| 41 | 41 | ||
| 42 | + | func TestSubagentStatusIncludesTurnCounters(t *testing.T) { | |
| 43 | + | entry := publicSubagentStatus(subagents.AgentSnapshot{ | |
| 44 | + | ID: "counter-agent", | |
| 45 | + | LifetimeTurns: 7, | |
| 46 | + | CurrentRunTurns: 2, | |
| 47 | + | }) | |
| 48 | + | if entry.LifetimeTurns != 7 || entry.CurrentRunTurns != 2 { | |
| 49 | + | t.Fatalf("status counters = (%d, %d), want (7, 2)", entry.LifetimeTurns, entry.CurrentRunTurns) | |
| 50 | + | } | |
| 51 | + | encoded, err := json.Marshal(entry) | |
| 52 | + | if err != nil { | |
| 53 | + | t.Fatal(err) | |
| 54 | + | } | |
| 55 | + | var fields map[string]any | |
| 56 | + | if err := json.Unmarshal(encoded, &fields); err != nil { | |
| 57 | + | t.Fatal(err) | |
| 58 | + | } | |
| 59 | + | if fields["lifetime_turns"] != float64(7) || fields["current_run_turns"] != float64(2) { | |
| 60 | + | t.Fatalf("encoded status counters = %#v", fields) | |
| 61 | + | } | |
| 62 | + | } | |
| 63 | + | ||
| 42 | 64 | func TestSubagentStatusSchemaHasOptionalAgentID(t *testing.T) { | |
| 43 | 65 | var schema struct { | |
| 44 | 66 | Type string `json:"type"` | |
| @@ -39,6 +39,28 @@ func (r statusTestRunner) Run(ctx context.Context, sink subagents.Sink) error { | |||||
| 39 | } | 39 | } | ||
| 40 | } | 40 | } | ||
| 41 | 41 | ||||
| 42 | + | func TestSubagentStatusIncludesTurnCounters(t *testing.T) { | |||
| 43 | + | entry := publicSubagentStatus(subagents.AgentSnapshot{ | |||
| 44 | + | ID: "counter-agent", | |||
| 45 | + | LifetimeTurns: 7, | |||
| 46 | + | CurrentRunTurns: 2, | |||
| 47 | + | }) | |||
| 48 | + | if entry.LifetimeTurns != 7 || entry.CurrentRunTurns != 2 { | |||
| 49 | + | t.Fatalf("status counters = (%d, %d), want (7, 2)", entry.LifetimeTurns, entry.CurrentRunTurns) | |||
| 50 | + | } | |||
| 51 | + | encoded, err := json.Marshal(entry) | |||
| 52 | + | if err != nil { | |||
| 53 | + | t.Fatal(err) | |||
| 54 | + | } | |||
| 55 | + | var fields map[string]any | |||
| 56 | + | if err := json.Unmarshal(encoded, &fields); err != nil { | |||
| 57 | + | t.Fatal(err) | |||
| 58 | + | } | |||
| 59 | + | if fields["lifetime_turns"] != float64(7) || fields["current_run_turns"] != float64(2) { | |||
| 60 | + | t.Fatalf("encoded status counters = %#v", fields) | |||
| 61 | + | } | |||
| 62 | + | } | |||
| 63 | + | ||||
| 42 | func TestSubagentStatusSchemaHasOptionalAgentID(t *testing.T) { | 64 | func TestSubagentStatusSchemaHasOptionalAgentID(t *testing.T) { | ||
| 43 | var schema struct { | 65 | var schema struct { | ||
| 44 | Type string `json:"type"` | 66 | Type string `json:"type"` | ||
M packages/core/session.go +10 -0
| @@ -1166,6 +1166,16 @@ func (s *Session) AppendMessage(m provider.Message) error { | |||
| 1166 | 1166 | return nil | |
| 1167 | 1167 | } | |
| 1168 | 1168 | ||
| 1169 | + | // Sync flushes the session file's bytes through the filesystem. Interactive | |
| 1170 | + | // hosts use this at acknowledgement boundaries where a crash must not make a | |
| 1171 | + | // prompt look accepted before its user message is recoverable. | |
| 1172 | + | func (s *Session) Sync() error { | |
| 1173 | + | if s == nil || s.writer == nil { | |
| 1174 | + | return nil | |
| 1175 | + | } | |
| 1176 | + | return s.writer.Sync() | |
| 1177 | + | } | |
| 1178 | + | ||
| 1169 | 1179 | // AppendCompaction writes a checkpoint that replaces all earlier | |
| 1170 | 1180 | // transcript rows when the session is resumed. The old rows remain in | |
| 1171 | 1181 | // the JSONL file for audit/export, while loaders use the latest | |
| @@ -1166,6 +1166,16 @@ func (s *Session) AppendMessage(m provider.Message) error { | |||||
| 1166 | return nil | 1166 | return nil | ||
| 1167 | } | 1167 | } | ||
| 1168 | 1168 | ||||
| 1169 | + | // Sync flushes the session file's bytes through the filesystem. Interactive | |||
| 1170 | + | // hosts use this at acknowledgement boundaries where a crash must not make a | |||
| 1171 | + | // prompt look accepted before its user message is recoverable. | |||
| 1172 | + | func (s *Session) Sync() error { | |||
| 1173 | + | if s == nil || s.writer == nil { | |||
| 1174 | + | return nil | |||
| 1175 | + | } | |||
| 1176 | + | return s.writer.Sync() | |||
| 1177 | + | } | |||
| 1178 | + | ||||
| 1169 | // AppendCompaction writes a checkpoint that replaces all earlier | 1179 | // AppendCompaction writes a checkpoint that replaces all earlier | ||
| 1170 | // transcript rows when the session is resumed. The old rows remain in | 1180 | // transcript rows when the session is resumed. The old rows remain in | ||
| 1171 | // the JSONL file for audit/export, while loaders use the latest | 1181 | // the JSONL file for audit/export, while loaders use the latest | ||