Merge pull request #120 from bnema/feat/subagent-live-observability-result-delivery
feat(subagents): make execution observable and durable
52 files · +2523 -512
M README.md +6 -6
| @@ -237,7 +237,7 @@ Print-mode stats contain `provider`, `model`, `prompt_tokens`, `reasoning_tokens | |||
| 237 | 237 | | `--no-tools` | Disable all tools. | | |
| 238 | 238 | | `--no-lsp` | Disable the built-in LSP/linter tool for this run. | | |
| 239 | 239 | | `--tools <csv>` | Only enable the listed tools. Include `lsp` to select it explicitly. An explicit list must include `web_search` to retain or opt into web search for that invocation; explicitly including it overrides the persisted web-search setting for that invocation. | | |
| 240 | - | | `--max-steps <n>` | Cap agent loop iterations (default 50). | | |
| 240 | + | | `--max-steps <n>` | Cap agent loop iterations (default: unlimited). | | |
| 241 | 241 | | `-e`, `--ext <path>` | Load an extension from `<path>` for this run (repeatable; wins against installed extensions of the same name). | | |
| 242 | 242 | | `--no-ext` | Skip extension discovery for this run. `--ext` still works on top, so `--no-ext --ext ./x` runs only `x`. | | |
| 243 | 243 | | `--no-skill` | Disable all skills, including built-ins. No `skill` tool is registered and the system prompt has no skill manifest. | | |
| @@ -272,7 +272,7 @@ When the sandbox is on (see `/jail`), filesystem tools and LSP workspace edits r | |||
| 272 | 272 | ||
| 273 | 273 | Workers inherit the parent CLI provider, model, and reasoning selection. Profile metadata from [docs/subagents.md](docs/subagents.md) is provided to the parent through `[subagents_list]` before selection; only the selected profile body and child-specific provider/model/reasoning configuration are loaded or applied afterward. Print mode writes only the final synthesis; stream mode renders every parent turn to stdout while tool diagnostics remain on stderr; JSON mode retains every parent event as parseable JSONL, including completion updates as existing `user_message` events. Worker output is evidence for the next parent turn, not a host log. | |
| 274 | 274 | ||
| 275 | - | The supervisor enforces configured concurrency, per-parent concurrency, queue/deadline, active-turn, output, and cancellation-grace limits; a headless invocation permits at most 32 completion follow-up waves. Cancellation stops the parent and workers through the normal graceful shutdown path. Workers cannot spawn descendants. See [docs/subagents.md](docs/subagents.md) for the profile manifest, limits, and unsupported combinations. | |
| 275 | + | The supervisor enforces configured concurrency, per-parent concurrency, queue/deadline, output, and cancellation-grace limits; a headless invocation permits at most 32 completion follow-up waves. Cancellation stops the parent and workers through the normal graceful shutdown path. Workers cannot spawn descendants. See [docs/subagents.md](docs/subagents.md) for the profile manifest, limits, and unsupported combinations. | |
| 276 | 276 | ||
| 277 | 277 | When an initial print, stream, or JSON request exceeds the provider context window, zut compacts the existing transcript and continues the already-appended prompt once. Print still writes only the recovered final text, stream keeps assistant text on stdout and tool diagnostics on stderr, and JSON suppresses the recoverable first terminal error so stdout remains JSONL for the successful turn. A compaction failure or a second context-window error remains terminal; this is not a general autonomous-follow-up policy. The same one-shot recovery applies to subagent workers. It does not apply to RPC, SDK, bot, or Telegram requests. | |
| 278 | 278 | ||
| @@ -293,7 +293,7 @@ Single-part names resolve only to a matching local directory or `.zut` archive. | |||
| 293 | 293 | ||
| 294 | 294 | Two ways to drive zut from another program: | |
| 295 | 295 | ||
| 296 | - | - **Go in-process**: import `github.com/bnema/zut/packages/agent/sdk`. One `Runtime` per project; `Prompt(ctx, text, images)` returns a channel of `Event`. Small example in `examples/sdk/`. | |
| 296 | + | - **Go in-process**: import `github.com/bnema/zut/packages/agent/sdk`. One `Runtime` per project; `Prompt(ctx, text, images)` returns a channel of `Event`. `sdk.Config.MaxSteps` is unlimited when zero; embedders preserving the former SDK default should set it to `50` explicitly. Small example in `examples/sdk/`. | |
| 297 | 297 | - **Any language, out-of-process**: spawn `zut rpc` as a subprocess and exchange newline-delimited JSON over its stdin/stdout. Wire format and event schema in [docs/rpc.md](docs/rpc.md). Reference clients live under `examples/rpc/`. | |
| 298 | 298 | ||
| 299 | 299 | Both interfaces share the same event schema, so transcripts captured by one can be replayed through the other. | |
| @@ -425,7 +425,7 @@ Background subagents that run alongside your main session. Each one is a separat | |||
| 425 | 425 | ||
| 426 | 426 | **`/session export` does NOT bundle subagents.** A `.zutsession` is just the main chat transcript; per-agent state (session file, unix-socket inbox) is machine-local and doesn't round-trip through a JSONL file. To share what an agent said, copy it out of the transcript view manually. | |
| 427 | 427 | ||
| 428 | - | **Subagent Orchestrator.** With `/settings` → **Subagent Orchestrator** on (or `/orchestrator`), the interactive primary agent enters strict orchestrator mode and gets whichever of the canonical `subagent_spawn` and `subagent_status` tools are permitted by launch-time policy. It must delegate all implementation, debugging/testing, and code-review work to an appropriately named profile, or to a clearly described general worker when no profile fits. It must not write or edit code, make direct implementation tool calls, inspect or review code, or apply worker patches itself. It may decompose requests, select and spawn workers, check status, coordinate follow-ups, and synthesize results. Give workers self-contained tasks: shared-worktree workers edit the same working directory and need coordination, while `isolation:"worktree"` is for parallel coding in separate trees; a worktree patch must be integrated by a worker rather than applied by the primary agent. Child workers cannot recursively spawn more sub-agents in v1. The `subagent_status` tool accepts an optional `agent_id`: omit it to list visible workers, or pass a full id / unique prefix to query one without waiting for completion. It returns bounded JSON metadata with the worker id, lifecycle state (`starting`, `running`, `completed`, `failed`, `cancelled`, or `detached`), underlying process/turn state, timestamps, a bounded first-line task summary, required state/target/unmet metadata when applicable, and terminal-result metadata when available; it omits full prompts, transcripts, result output, provider settings, and filesystem paths. Treat the bounded task summary as session data. The spawn tool always returns the child id and initial state immediately. Set `required:true` when the parent or workflow depends on the result: the worker remains asynchronous, its outcome is persisted and delivered through a host completion update, and failure, timeout, or cancellation remains recoverable until a retry succeeds or the user explicitly removes the terminal worker. An outcome not durably observed before host restart becomes indeterminate and requires the user to inspect side effects before explicitly resuming, restarting, or removing the worker. Completion is a durable `turn.result` with a `subagent://<id>/result` reference. The tool accepts named profiles, timeout, `max_turns`, and shared/worktree isolation. Omit `max_turns` to use the supervisor policy; the default ceiling is 3 and `subagents.max_turns` changes it. Global concurrency limits apply equally to slash commands, tool calls, and batches. Completion is host-event-driven for every worker through `[auto-subagents update]`, with required state also supplied as high-priority `[required-subagents update]` context. Never use `bash sleep`, `watch`, `tail -f`, polling loops, repeated `subagent_status`, or dashboard/metadata/event-log/file checks solely to wait; work on unrelated independent tasks or end/yield your turn. Legitimate waits inside user-requested commands, provider flows, extensions, or tests remain allowed. Off by default; toggle with `/orchestrator` or from `/settings`. When off, the permitted canonical subagent tools and profile manifest remain available for user-requested delegation or when an active skill workflow requires it; the primary otherwise performs work itself without the strict orchestrator contract. | |
| 428 | + | **Subagent Orchestrator.** With `/settings` → **Subagent Orchestrator** on (or `/orchestrator`), the interactive primary agent enters strict orchestrator mode and gets whichever of the canonical `subagent_spawn` and `subagent_status` tools are permitted by launch-time policy. It must delegate all implementation, debugging/testing, and code-review work to an appropriately named profile, or to a clearly described general worker when no profile fits. It must not write or edit code, make direct implementation tool calls, inspect or review code, or apply worker patches itself. It may decompose requests, select and spawn workers, check status, coordinate follow-ups, and synthesize results. Give workers self-contained tasks: shared-worktree workers edit the same working directory and need coordination, while `isolation:"worktree"` is for parallel coding in separate trees; a worktree patch must be integrated by a worker rather than applied by the primary agent. Child workers cannot recursively spawn more sub-agents in v1. The `subagent_status` tool accepts an optional `agent_id`: omit it to list visible workers, or pass a full id / unique prefix to query one without waiting for completion. It returns bounded factual metadata: the worker id, current primary operation, last safe observation/event, terminal fact, timestamps, turn counters, a bounded first-line task summary, requirement state, and terminal-result delivery metadata. It omits full prompts, transcripts, result output, provider settings, and filesystem paths. Treat the bounded task summary as session data. The spawn tool always returns the child id and initial state immediately. Set `required:true` when the parent or workflow depends on the result: the worker remains asynchronous, its outcome is persisted and delivered through a host completion update, and failure, timeout, or cancellation remains recoverable until a retry succeeds or the user explicitly removes the terminal worker. An outcome not durably observed before host restart becomes indeterminate and requires the user to inspect side effects before explicitly resuming, restarting, or removing the worker. Completion is a durable `turn.result` with a `subagent://<id>/result` reference. The tool accepts named profiles and shared/worktree isolation. Model steps and delegated turns are counted for observability but are never capped; only an optional manager-owned `subagents.default_timeout` bounds active turn duration. Global concurrency limits apply equally to slash commands, tool calls, and batches. Completion is host-event-driven for every worker through `[auto-subagents update]`, with required state also supplied as high-priority `[required-subagents update]` context. Never use `bash sleep`, `watch`, `tail -f`, polling loops, repeated `subagent_status`, or dashboard/metadata/event-log/file checks solely to wait; work on unrelated independent tasks or end/yield your turn. Legitimate waits inside user-requested commands, provider flows, extensions, or tests remain allowed. Off by default; toggle with `/orchestrator` or from `/settings`. When off, the permitted canonical subagent tools and profile manifest remain available for user-requested delegation or when an active skill workflow requires it; the primary otherwise performs work itself without the strict orchestrator contract. | |
| 429 | 429 | ||
| 430 | 430 | ### Named subagent profiles | |
| 431 | 431 | ||
| @@ -439,7 +439,7 @@ Opens a dialog with every persistent setting. `up`/`down` to navigate, `enter` o | |||
| 439 | 439 | - **render images when supported** — draw screenshots / `read`-returned images inline using the terminal's image protocol, or fall back to a text placeholder. Auto-detected from `TERM_PROGRAM`; the toggle overrides the detection. The row is greyed out and forced off on terminals that don't speak any image protocol. | |
| 440 | 440 | - **terminal alerts** — emit a terminal bell when the main session stops with work that may need attention, or when an extension raises a structured alert. Enabled by default; changes apply immediately and persist as `terminal_alerts_enabled`. Terminal emulators may render the bell audibly, visually, or not at all. | |
| 441 | 441 | - **AI terminal titles** — after the first real prompt of a fresh interactive session, make one small hidden request to the active model and set the terminal title to `zut: <title>`. The title is limited to 40 Unicode characters, persisted with the session, restored on resume, and never added to the conversation. Enabled by default; disable it to avoid the extra model request. The toggle applies immediately, but title generation still waits for the first real prompt and never starts from startup context, resumed history, or slash commands. Persists as `terminal_title_enabled`. | |
| 442 | - | - **Subagent Orchestrator** — enable strict primary-agent orchestrator mode: delegate implementation, debugging/testing, and code-review work to named or general workers; the primary agent coordinates but does not make direct implementation changes. It can spawn optional or required background sub-agents via `subagent_spawn` and query live state via `subagent_status` when those tools are permitted. If delegation is unavailable because of launch-time restrictions, the contract remains active, profile metadata is withheld, and the primary reports the limitation rather than implementing directly. Off by default. When off, the permitted canonical subagent tools and profile manifest remain available for user-requested delegation or when an active skill workflow requires it; the primary otherwise works directly without the strict orchestrator contract. The tools accept named profiles, reasoning, per-spawn fast-mode overrides, timeout, turn limits, and shared/worktree isolation where applicable; lifecycle and result references are persisted. Completion is host-event-driven through `[auto-subagents update]` for every worker; required work stays asynchronous, persists its outcome, and prevents only the terminal parent response until satisfied or explicitly waived by user removal. Do not poll or inspect dashboards, metadata, event logs, or files solely to wait. See `/subagents` and [docs/subagents.md](docs/subagents.md) for details. | |
| 442 | + | - **Subagent Orchestrator** — enable strict primary-agent orchestrator mode: delegate implementation, debugging/testing, and code-review work to named or general workers; the primary agent coordinates but does not make direct implementation changes. It can spawn optional or required background sub-agents via `subagent_spawn` and query live state via `subagent_status` when those tools are permitted. If delegation is unavailable because of launch-time restrictions, the contract remains active, profile metadata is withheld, and the primary reports the limitation rather than implementing directly. Off by default. When off, the permitted canonical subagent tools and profile manifest remain available for user-requested delegation or when an active skill workflow requires it; the primary otherwise works directly without the strict orchestrator contract. The tools accept named profiles, reasoning, per-spawn fast-mode overrides, and shared/worktree isolation where applicable; lifecycle and result references are persisted. Completion is host-event-driven through `[auto-subagents update]` for every worker; required work stays asynchronous, persists its outcome, and prevents only the terminal parent response until satisfied or explicitly waived by user removal. Do not poll or inspect dashboards, metadata, event logs, or files solely to wait. See `/subagents` and [docs/subagents.md](docs/subagents.md) for details. | |
| 443 | 443 | - **Ponytail coding mode** — include compact engineering guidance in each resolved system prompt; the guidance tells the model to apply it to coding, debugging, and review work rather than ordinary conversation. It favors understanding the real flow, small validated changes, reuse, and preserving safety checks. Enabled by default; changes apply to the next model call and persist as `ponytail_enabled` in `$ZUT_HOME/config.json`. It is included by interactive, print, stream, JSON, RPC, subagent, bot, SDK, and Zutfile agents. | |
| 444 | 444 | - **fast mode** — request the provider's fast tier where supported. It currently uses OpenAI's Fast service tier for OpenAI, OpenAI Responses, and OpenAI Codex models. Off by default; toggle it with `/fast` or `/settings`; changes apply on the next model call and persist as `fast_mode`. Other providers currently return an unsupported-provider error. Fast mode may cost more and depends on the selected model/account. | |
| 445 | 445 | - **lsp in main session** — enable the built-in `lsp` tool and code diagnostics for the main agent. Enabled by default; changes persist as `lsp_enabled`. | |
| @@ -502,7 +502,7 @@ When a turn fails because of a recoverable provider error — expired token (`40 | |||
| 502 | 502 | ||
| 503 | 503 | The picker is the same vertical list / fuzzy filter UI as `/model`, but it only shows models from providers you're currently logged in to (env vars, `auth.json`, Kimi CLI fallback, ollama). The failed model is excluded. Press `↑`/`↓` to choose, `enter` to retry the **same prompt** on the new model, `esc` to dismiss. | |
| 504 | 504 | ||
| 505 | - | Before the actual provider request fires, the OpenAI / Anthropic / Kimi / DeepSeek / Google / OpenAI-Codex clients also do up to two silent retries with short backoff (250ms, 750ms) on `502`/`503`/`504` and connection-reset / EOF-before-headers errors. Most edge-proxy blips disappear without you ever seeing the rescue picker. | |
| 505 | + | Before the actual provider request fires, the shared HTTP streaming clients used by OpenAI / Anthropic / Kimi / DeepSeek / Google / OpenAI-Codex also do up to four silent retries with short backoff (250ms, then 750ms, 750ms, and 750ms) on transient HTTP failures and connection-reset / EOF-before-headers errors. Once a stream is open, five reconnect attempts and a five-minute idle timeout independently guard stalled or interrupted streams. Most edge-proxy blips disappear without you ever seeing the rescue picker. | |
| 506 | 506 | ||
| 507 | 507 | A rescue retry always **drops launch-time `--api-key` and `--base-url`** before rebuilding the agent. Those overrides are usually the reason the rescue triggered (bad key, typo'd base URL, corporate gateway only valid for the originally-picked provider), so the retry re-resolves credentials from env vars / `auth.json` / provider defaults instead. Use `/model` if you want overrides to stick. | |
| 508 | 508 | ||
M docs/subagents.md +7 -7
| @@ -62,7 +62,7 @@ The parent CLI's `--provider`, `--model`, and `--reasoning` values are inherited | |||
| 62 | 62 | zut -p --orchestrate --provider openai --model gpt-5 --reasoning high "delegate the implementation and synthesize the result" | |
| 63 | 63 | ``` | |
| 64 | 64 | ||
| 65 | - | The parent runs completion-driven waves rather than polling, with at most 32 follow-up waves per invocation. Configured concurrency and per-parent concurrency limits, queue and turn deadlines, maximum turns/output, and the graceful cancellation/shutdown period apply. Cancellation propagates to the parent and workers through the supervisor; partial worker evidence remains available when the worker reports it. | |
| 65 | + | The parent runs completion-driven waves rather than polling, with at most 32 follow-up waves per invocation. Configured concurrency and per-parent concurrency limits, queue and turn deadlines, output, and the graceful cancellation/shutdown period apply. Cancellation propagates to the parent and workers through the supervisor; partial worker evidence remains available when the worker reports it. | |
| 66 | 66 | ||
| 67 | 67 | Output follows the selected existing mode. Print writes only the final synthesis. Stream renders every parent turn on stdout and keeps tool diagnostics on stderr. JSON retains every event from every parent turn as parseable JSONL; completion reports are injected as ordinary `user_message` events, and a failed primary or handoff produces one terminal JSON error object rather than a host log. | |
| 68 | 68 | ||
| @@ -137,9 +137,9 @@ Shared mode preserves the historical host working directory, so workers there mu | |||
| 137 | 137 | ||
| 138 | 138 | Every child retains narrowly scoped internal process control and durable task/result metadata. These details coordinate startup, cancellation, and recovery; they are not a user-facing progress model. Diagnostic progress comes only from paired trace boundaries and their projection into open operations, terminal facts, or an explicitly insufficient observation. | |
| 139 | 139 | ||
| 140 | - | During a trace-observed open operation, interactive mode shows compact rows beside the input area: a themed spinner, the named profile (or agent ID), the open operation, and its elapsed duration. Rows default below the input and can move above it through **running subagent position** in `/settings` → **tui settings**. On short terminals, omitted operations collapse into one count summary so the editor remains visible. A process lifecycle, heartbeat, or generic `working` label never creates a row; use `/subagents` for the current factual operation or last observed fact. | |
| 140 | + | During a trace-observed open operation, interactive mode shows compact rows beside the input area: a themed spinner, the named profile (or agent ID), the most specific open operation, and its elapsed duration. The trace keeps the active turn boundary even after older raw stream fragments leave its bounded in-memory window, so a long-running worker remains visible and its elapsed timer stays anchored to the delegated turn. A recent safe observation is shown alongside it when available, for example `assistant streaming 1s ago`; normal mode never displays message text, reasoning text, tool arguments, or tool output. Rows default below the input and can move above it through **running subagent position** in `/settings` → **tui settings**. On short terminals, omitted operations collapse into one count summary so the editor remains visible. A process lifecycle, heartbeat, or generic `working` label never creates a row; use `/subagents` for the current factual operation or last observed fact. | |
| 141 | 141 | ||
| 142 | - | `/subagents` opens on workers from the current host session. Press **Tab** to switch to the all-sessions history, use **Up/Down** to select a worker, and press **Right** or **l** to expand a bounded live snapshot of its user messages, assistant output, and tool activity (**Left** or **h** collapses it). The dashboard derives its row budget from the terminal height, keeping the selected worker visible on short terminals. Press **Enter** for the complete transcript. | |
| 142 | + | `/subagents` opens on workers from the current host session. Press **Tab** to switch to the all-sessions history, use **Up/Down** to select a worker, and press **Right** or **l** to expand a bounded live snapshot of its user messages, assistant output, and tool activity (**Left** or **h** collapses it). The dashboard derives its row budget from the terminal height, keeping the selected worker visible on short terminals. Press **Enter** for the complete transcript. `/subagents wait <id>` observes the worker's current delegated turn, not its long-lived daemon process. Canceling that wait only removes the observer; it never stops the worker. | |
| 143 | 143 | ||
| 144 | 144 | The worker and supervisor communicate over newline-delimited JSON. Current messages use version `1` envelopes with `version`, `message_id`, `agent_id`, `turn_id`, `timestamp`, and `payload`. Unknown event names and payload fields are retained. Only versioned JSONL envelopes are accepted on the worker protocol. | |
| 145 | 145 | ||
| @@ -156,11 +156,11 @@ subagent://<id>/patch | |||
| 156 | 156 | ||
| 157 | 157 | Workers send protocol events to their parent supervisor. The supervisor maintains the live transcript and, when execution tracing is enabled, writes one ordered local trace for the parent/child tree. The trace is diagnostic data, not a second session transcript. | |
| 158 | 158 | ||
| 159 | - | A subagent timeout applies to each active delegated turn. When the deadline expires, the worker cancels that turn, writes a failed result with error code `deadline_exceeded`, preserves any output already received, and remains idle so an explicit follow-up can resume its session. Idle time does not consume the next turn's timeout, and each resumed turn receives a fresh deadline. Process shutdown still allows the configured grace period for the worker to write its session and result before forceful cancellation. | |
| 159 | + | Delegated turns have no wall-clock deadline by default. A positive configured `default_timeout` applies only to the active delegated turn; timeout selection is manager-owned and is not exposed to the model-facing spawn tool. When that deadline expires, the worker cancels the turn, writes a failed result with error code `deadline_exceeded`, preserves any output already received, and remains idle so an explicit follow-up can resume its session. Idle time does not consume the next turn's timeout, and each resumed turn receives a fresh deadline. Provider stream-idle detection and retry limits remain independently bounded, so an unlimited turn does not mean a silent connection can hang forever. Process shutdown still allows the configured grace period for the worker to write its session and result before forceful cancellation. | |
| 160 | 160 | ||
| 161 | 161 | Use `/subagents resume-session <id>` to continue the existing session without replaying its original task. Use `/subagents restart-task <id>` only when intentionally starting the stored task again. Cancellation requests graceful shutdown first, then forcefully cancels after the configured grace period. | |
| 162 | 162 | ||
| 163 | - | When the relevant manager tool is permitted by launch-time policy, the model can manage workers with `subagent_status`, `subagent_stop`, and `subagent_resume`. This includes explicit user-requested delegation and required-work recovery when auto-subagents orchestration is disabled. Call `subagent_status` with `{}` to list workers visible to the active supervisor session, or pass `{"agent_id":"<id-or-unique-prefix>"}` to query one worker. Queries use the shared trace projection and never wait for a worker turn or process to finish; repeated calls or state inspection are not completion signals. The JSON response contains the worker id, open operation and its start time when one is observed, a terminal fact or last observed event, `lifetime_turns` and `current_run_turns`, a bounded first-line task summary, required state/target/unmet/error-code metadata when applicable, and terminal-result metadata/reference when available. It deliberately omits generic lifecycle claims such as `working`, `alive`, `idle`, `process_state`, and `turn_state`, as well as prompts, transcripts, result output, provider settings, and filesystem paths. The bounded task summary can reflect user task text and must be treated as session data. | |
| 163 | + | When the relevant manager tool is permitted by launch-time policy, the model can manage workers with `subagent_status`, `subagent_stop`, and `subagent_resume`. This includes explicit user-requested delegation and required-work recovery when auto-subagents orchestration is disabled. Call `subagent_status` with `{}` to list workers visible to the active supervisor session, or pass `{"agent_id":"<id-or-unique-prefix>"}` to query one worker. Queries use the shared trace projection and never wait for a worker turn or process to finish; repeated calls or state inspection are not completion signals. The JSON response contains the worker id, open operation and its start time when one is observed, the derived `primary_operation`, the last safe live observation and timestamp, a terminal fact or last event, `lifetime_turns` and `current_run_turns`, a bounded first-line task summary, required state/target/unmet/error-code metadata when applicable, and terminal-result availability/reference/delivery metadata when available. A result is only marked delivered after the parent accepts its required-work notification. It deliberately omits generic lifecycle claims such as `working`, `alive`, `idle`, `process_state`, and `turn_state`, as well as prompts, transcripts, result output, provider settings, and filesystem paths. The bounded task summary can reflect user task text and must be treated as session data. | |
| 164 | 164 | ||
| 165 | 165 | ## Local execution trace | |
| 166 | 166 | ||
| @@ -174,8 +174,8 @@ zut debug trace <bundle> | |||
| 174 | 174 | ||
| 175 | 175 | Start with the oldest open operation shown by the command. A provider request, tool invocation, or turn is useful evidence; a process that is merely alive without an open operation is reported as `no observable operation` with its last fact, not as progress. | |
| 176 | 176 | ||
| 177 | - | Use `subagent_stop` with an `agent_id` to request termination of a stuck worker through the normal graceful-then-forceful supervisor lifecycle. Its `stop_requested` result confirms that shutdown has started; wait for the host completion update before attempting a restart. Use `subagent_resume` with an `agent_id` and `prompt` to start a fresh max-turn budget while retaining the transcript/session/context, send the follow-up to an idle worker, queue it for the next available message turn when the worker is active, or restart a terminal worker from its retained session with that follow-up as the initial turn. Set `required:true` to make a new follow-up mandatory. A resume of failed, timed-out, or canceled required work is mandatory automatically, returns immediately, reports its new terminal outcome asynchronously, and advances the persisted target turn. An indeterminate outcome requires the explicit user reconciliation described above before a new target turn starts. A follow-up remains durably pending until its matching user message is flushed to the session and the delegated `turn.started` boundary is recorded; a worker-ready event alone does not acknowledge it. Automatic retries, model loops, and compaction remain within the current run budget; multiple active follow-ups are delivered in order. Only same-worker `turn.start` retries have stable in-memory command deduplication; that deduplication is lost when the worker restarts. Do not retry `turn.cancel`, `agent.shutdown`, or `agent.ping` after an ambiguous outcome. After a worker restart, explicitly reconcile durable results and side effects before retrying non-idempotent work. It never replays the original task. Optional and required completion use the normal auto-subagent update; required state is additionally supplied in the required-update context before the parent can finish. An unknown or ambiguous id is returned as a model-visible tool error; the existing `/subagents` dashboard and final-result notifications remain unchanged. An explicit `--tools` allowlist must include each manager tool that the model needs, and `subagent_status` must not be polled solely to wait for completion. | |
| 177 | + | Use `subagent_stop` with an `agent_id` to request termination of a stuck worker through the normal graceful-then-forceful supervisor lifecycle. Its `stop_requested` result confirms that shutdown has started; wait for the host completion update before attempting a restart. Use `subagent_resume` with an `agent_id` and `prompt` to start a fresh observable run while retaining the transcript/session/context, send the follow-up to an idle worker, queue it for the next available message turn when the worker is active, or restart a terminal worker from its retained session with that follow-up as the initial turn. Set `required:true` to make a new follow-up mandatory. A resume of failed, timed-out, or canceled required work is mandatory automatically, returns immediately, reports its new terminal outcome asynchronously, and advances the persisted target turn. An indeterminate outcome requires the explicit user reconciliation described above before a new target turn starts. A follow-up remains durably pending until its matching user message is flushed to the session and the delegated `turn.started` boundary is recorded; a worker-ready event alone does not acknowledge it. Automatic retries, model loops, and compaction continue within the current run; multiple active follow-ups are delivered in order. Only same-worker `turn.start` retries have stable in-memory command deduplication; that deduplication is lost when the worker restarts. Do not retry `turn.cancel`, `agent.shutdown`, or `agent.ping` after an ambiguous outcome. After a worker restart, explicitly reconcile durable results and side effects before retrying non-idempotent work. It never replays the original task. Optional and required completion use the normal auto-subagent update; required state is additionally supplied in the required-update context before the parent can finish. An unknown or ambiguous id is returned as a model-visible tool error; the existing `/subagents` dashboard and final-result notifications remain unchanged. An explicit `--tools` allowlist must include each manager tool that the model needs, and `subagent_status` must not be polled solely to wait for completion. | |
| 178 | 178 | ||
| 179 | 179 | ## Resource policy | |
| 180 | 180 | ||
| 181 | - | The persisted `subagents` config object supports `max_concurrent`, `max_concurrent_per_parent`, `queue_timeout`, `startup_timeout`, `default_timeout`, `max_turns`, output caps, allowed tools/roots, heartbeat and idle timeouts. A worker fails with an actionable queue-timeout error when no execution slot becomes available before `queue_timeout` (5 minutes by default). Once admitted, it must report `agent_ready` before `startup_timeout` (1 minute by default) or fails with guidance to inspect worker output and retry. `max_turns` bounds one active worker run; lifetime and current-run turn counters remain persisted and visible in worker status. An explicit `subagent_resume` starts a new run with a fresh budget without replaying the original task or losing session context. Previously saved `max_total_spawned` values are accepted but ignored. A missing or non-positive configured `max_turns` uses the default ceiling of 3. The `max_turns` field in `subagent_spawn` is optional: omit it to use the policy ceiling, or provide a value from 1 through the configured maximum. Concurrency limits apply to slash commands, `subagent_spawn`, and batch operations. A child cannot create descendants in v1. Per-agent timeouts are retained across reload/resume. Packaged `zut run` agents keep their declared capability ceiling by disabling subagent delegation; profiles are not a substitute for that permission boundary. A primary whose launch-time policy disables delegation still receives the strict orchestrator contract, but does not receive actionable profile metadata or a replacement implementation path. | |
| 181 | + | The persisted `subagents` config object supports `max_concurrent`, `max_concurrent_per_parent`, `queue_timeout`, `startup_timeout`, `default_timeout`, output caps, allowed tools/roots, heartbeat and idle timeouts. Duration strings use Go duration syntax and malformed values fail configuration loading with the field name. A worker fails with an actionable queue-timeout error when no execution slot becomes available before `queue_timeout` (5 minutes by default). Once admitted, it must report `agent_ready` before `startup_timeout` (1 minute by default) or fails with guidance to inspect worker output and retry. Model steps and delegated turns are counted for observability but never capped. A missing or non-positive `default_timeout` leaves each active delegated turn unlimited; a positive value opts into a manager-owned deadline retained across reload and resume. An explicit `subagent_resume` starts a new observable run without replaying the original task or losing session context. Previously saved `max_total_spawned` values are accepted but ignored. Concurrency limits apply to slash commands, `subagent_spawn`, and batch operations. A child cannot create descendants in v1. The shared HTTP streaming providers retry transient request-open failures four times; after opening, a stream may reconnect five times and is canceled after five minutes without an event. Packaged `zut run` agents keep their declared capability ceiling by disabling subagent delegation; profiles are not a substitute for that permission boundary. A primary whose launch-time policy disables delegation still receives the strict orchestrator contract, but does not receive actionable profile metadata or a replacement implementation path. | |
M packages/agent/args.go +0 -12
| @@ -147,8 +147,6 @@ type Args struct { | |||
| 147 | 147 | // --subagent-worker. | |
| 148 | 148 | SubagentWorker string | |
| 149 | 149 | ||
| 150 | - | // SubagentMaxTurns limits message turns in one worker run. | |
| 151 | - | SubagentMaxTurns int | |
| 152 | 150 | // SubagentTurnTimeout limits each active delegated turn while leaving the | |
| 153 | 151 | // worker process alive between turns for explicit follow-ups. | |
| 154 | 152 | SubagentTurnTimeout time.Duration | |
| @@ -387,16 +385,6 @@ func ParseArgs(in []string) (Args, error) { | |||
| 387 | 385 | return a, fmt.Errorf("--max-steps must be a positive integer") | |
| 388 | 386 | } | |
| 389 | 387 | a.MaxSteps = n | |
| 390 | - | case "--max-turns": | |
| 391 | - | v, err := want(&i, arg) | |
| 392 | - | if err != nil { | |
| 393 | - | return a, err | |
| 394 | - | } | |
| 395 | - | var n int | |
| 396 | - | if _, err := fmt.Sscanf(v, "%d", &n); err != nil || n <= 0 { | |
| 397 | - | return a, fmt.Errorf("--max-turns must be a positive integer") | |
| 398 | - | } | |
| 399 | - | a.SubagentMaxTurns = n | |
| 400 | 388 | case "--subagent-turn-timeout": | |
| 401 | 389 | v, err := want(&i, arg) | |
| 402 | 390 | if err != nil { | |
M packages/agent/config.go +28 -1
| @@ -30,7 +30,6 @@ type SubagentsConfig struct { | |||
| 30 | 30 | QueueTimeout string `json:"queue_timeout,omitempty"` | |
| 31 | 31 | StartupTimeout string `json:"startup_timeout,omitempty"` | |
| 32 | 32 | DefaultTimeout string `json:"default_timeout,omitempty"` | |
| 33 | - | MaxTurns int `json:"max_turns,omitempty"` | |
| 34 | 33 | MaxOutputBytes int `json:"max_output_bytes,omitempty"` | |
| 35 | 34 | MaxOutputLines int `json:"max_output_lines,omitempty"` | |
| 36 | 35 | AllowedTools []string `json:"allowed_tools,omitempty"` | |
| @@ -321,9 +320,37 @@ func LoadConfig() (Config, error) { | |||
| 321 | 320 | if err := json.Unmarshal(b, &c); err != nil { | |
| 322 | 321 | return c, fmt.Errorf("parse config: %w", err) | |
| 323 | 322 | } | |
| 323 | + | if err := validateSubagentDurations(c.Subagents); err != nil { | |
| 324 | + | return c, err | |
| 325 | + | } | |
| 324 | 326 | return c, nil | |
| 325 | 327 | } | |
| 326 | 328 | ||
| 329 | + | func validateSubagentDurations(cfg SubagentsConfig) error { | |
| 330 | + | values := []struct { | |
| 331 | + | name string | |
| 332 | + | value string | |
| 333 | + | }{ | |
| 334 | + | {"queue_timeout", cfg.QueueTimeout}, | |
| 335 | + | {"startup_timeout", cfg.StartupTimeout}, | |
| 336 | + | {"default_timeout", cfg.DefaultTimeout}, | |
| 337 | + | {"heartbeat_interval", cfg.HeartbeatInterval}, | |
| 338 | + | {"idle_timeout", cfg.IdleTimeout}, | |
| 339 | + | {"reconnect_timeout", cfg.ReconnectTimeout}, | |
| 340 | + | {"cancel_grace_period", cfg.CancelGracePeriod}, | |
| 341 | + | } | |
| 342 | + | for _, item := range values { | |
| 343 | + | value := strings.TrimSpace(item.value) | |
| 344 | + | if value == "" { | |
| 345 | + | continue | |
| 346 | + | } | |
| 347 | + | if _, err := time.ParseDuration(value); err != nil { | |
| 348 | + | return fmt.Errorf("parse config: subagents.%s: %w", item.name, err) | |
| 349 | + | } | |
| 350 | + | } | |
| 351 | + | return nil | |
| 352 | + | } | |
| 353 | + | ||
| 327 | 354 | // SaveConfig writes the config file, creating parent dirs. | |
| 328 | 355 | func SaveConfig(c Config) error { | |
| 329 | 356 | if err := os.MkdirAll(ZutHome(), 0o755); err != nil { | |
M packages/agent/modes/subagent_indicator.go +48 -7
| @@ -10,6 +10,11 @@ import ( | |||
| 10 | 10 | "github.com/mattn/go-runewidth" | |
| 11 | 11 | ) | |
| 12 | 12 | ||
| 13 | + | var providerWaitingSpinner = tui.NewStringSpinner( | |
| 14 | + | []string{"Zzzz", "zZzz", "zzZz", "zzzZ"}, | |
| 15 | + | time.Second, | |
| 16 | + | ) | |
| 17 | + | ||
| 13 | 18 | // renderSubagentActivityLines renders only trace-observed open operations. | |
| 14 | 19 | // A living process, generic lifecycle state, or heartbeat never starts a | |
| 15 | 20 | // spinner because none proves that an operation is progressing. | |
| @@ -17,10 +22,11 @@ func renderSubagentActivityLines(th tui.Theme, spinnerGlyph string, snapshots [] | |||
| 17 | 22 | lines := make([]string, 0, len(snapshots)) | |
| 18 | 23 | for _, snapshot := range snapshots { | |
| 19 | 24 | view := views[snapshot.ID] | |
| 20 | - | if len(view.OpenOperations) == 0 { | |
| 25 | + | operation, ok := view.Primary() | |
| 26 | + | if !ok { | |
| 21 | 27 | continue | |
| 22 | 28 | } | |
| 23 | - | line := renderSubagentActivityLine(th, spinnerGlyph, snapshot, view.OpenOperations[0], width, now) | |
| 29 | + | line := renderSubagentActivityLine(th, spinnerGlyph, snapshot, operation, view.TurnStartedAt(operation), view.ObservationFor(operation), width, now) | |
| 24 | 30 | if line != "" { | |
| 25 | 31 | lines = append(lines, line) | |
| 26 | 32 | } | |
| @@ -28,7 +34,7 @@ func renderSubagentActivityLines(th tui.Theme, spinnerGlyph string, snapshots [] | |||
| 28 | 34 | return lines | |
| 29 | 35 | } | |
| 30 | 36 | ||
| 31 | - | func renderSubagentActivityLine(th tui.Theme, spinnerGlyph string, snapshot subagents.AgentSnapshot, operation subagents.Operation, width int, now time.Time) string { | |
| 37 | + | func renderSubagentActivityLine(th tui.Theme, spinnerGlyph string, snapshot subagents.AgentSnapshot, operation subagents.Operation, turnStartedAt time.Time, observation *subagents.LiveObservation, width int, now time.Time) string { | |
| 32 | 38 | name := sanitizeSubagentIndicatorText(snapshot.Subagent) | |
| 33 | 39 | if name == "" { | |
| 34 | 40 | name = sanitizeSubagentIndicatorText(snapshot.ID) | |
| @@ -40,8 +46,24 @@ func renderSubagentActivityLine(th tui.Theme, spinnerGlyph string, snapshot suba | |||
| 40 | 46 | if spinnerGlyph == "" { | |
| 41 | 47 | spinnerGlyph = "." | |
| 42 | 48 | } | |
| 43 | - | activity := operation.Label() | |
| 44 | - | age := formatSubagentActivityAge(operation.StartedAt, now) | |
| 49 | + | activity := subagentOperationActivity(operation, now) | |
| 50 | + | if observation != nil { | |
| 51 | + | activity += " · " + observation.Label() + " " + formatSubagentActivityAge(observation.At, now) + " ago" | |
| 52 | + | } | |
| 53 | + | // The rightmost age is elapsed time for the open delegated turn, stable | |
| 54 | + | // across its nested provider/tool cycles. Fall back to the operation start | |
| 55 | + | // only for legacy/incomplete traces without a matching turn boundary. | |
| 56 | + | if turnStartedAt.IsZero() { | |
| 57 | + | turnStartedAt = operation.StartedAt | |
| 58 | + | } | |
| 59 | + | age := formatSubagentActivityAge(turnStartedAt, now) | |
| 60 | + | if operation.Step > 0 { | |
| 61 | + | label := "steps" | |
| 62 | + | if operation.Step == 1 { | |
| 63 | + | label = "step" | |
| 64 | + | } | |
| 65 | + | age = fmt.Sprintf("%d %s · %s", operation.Step, label, age) | |
| 66 | + | } | |
| 45 | 67 | plain, layout := fitSubagentActivityLine(spinnerGlyph, name, activity, age, width-2) | |
| 46 | 68 | if plain == "" { | |
| 47 | 69 | return "" | |
| @@ -56,7 +78,7 @@ func renderSubagentActivityLine(th tui.Theme, spinnerGlyph string, snapshot suba | |||
| 56 | 78 | name = strings.TrimPrefix(parts[0], spinnerGlyph+" ") | |
| 57 | 79 | return " " + th.FGColor(th.Spinner, spinnerGlyph) + " " + | |
| 58 | 80 | th.FGColor(th.Assistant, name) + th.FGColor(th.Muted, " · ") + | |
| 59 | - | th.FGColor(th.FG, parts[1]) + th.FGColor(th.Muted, " · "+parts[2]) | |
| 81 | + | th.FGColor(subagentOperationColor(th, operation), parts[1]) + th.FGColor(th.Muted, " · "+parts[2]) | |
| 60 | 82 | case subagentActivityNameAge: | |
| 61 | 83 | parts := strings.SplitN(plain, " · ", 2) | |
| 62 | 84 | if len(parts) != 2 { | |
| @@ -70,6 +92,24 @@ func renderSubagentActivityLine(th tui.Theme, spinnerGlyph string, snapshot suba | |||
| 70 | 92 | } | |
| 71 | 93 | } | |
| 72 | 94 | ||
| 95 | + | func subagentOperationActivity(operation subagents.Operation, now time.Time) string { | |
| 96 | + | if operation.Type == "provider.request.started" { | |
| 97 | + | return providerWaitingSpinner.FrameAt(operation.StartedAt, now) | |
| 98 | + | } | |
| 99 | + | return operation.Label() | |
| 100 | + | } | |
| 101 | + | ||
| 102 | + | func subagentOperationColor(th tui.Theme, operation subagents.Operation) tui.TerminalColor { | |
| 103 | + | switch operation.Type { | |
| 104 | + | case "provider.request.started": | |
| 105 | + | return th.Spinner | |
| 106 | + | case "tool.started": | |
| 107 | + | return th.Tool | |
| 108 | + | default: | |
| 109 | + | return th.Assistant | |
| 110 | + | } | |
| 111 | + | } | |
| 112 | + | ||
| 73 | 113 | type subagentActivityLayout uint8 | |
| 74 | 114 | ||
| 75 | 115 | const ( | |
| @@ -167,7 +207,8 @@ func (i *Interactive) activeSubagentActivitySnapshots() ([]subagents.AgentSnapsh | |||
| 167 | 207 | if activeSession != "" && agent.SessionID != "" && agent.SessionID != activeSession { | |
| 168 | 208 | continue | |
| 169 | 209 | } | |
| 170 | - | if len(views[agent.ID].OpenOperations) == 0 { | |
| 210 | + | view := views[agent.ID] | |
| 211 | + | if _, ok := view.Primary(); !ok { | |
| 171 | 212 | continue | |
| 172 | 213 | } | |
| 173 | 214 | out = append(out, subagents.AgentSnapshot{ID: agent.ID, Started: agent.Started, Subagent: agent.Subagent}) | |
M packages/agent/modes/subagent_indicator_test.go +79 -1
| @@ -21,12 +21,77 @@ func TestRenderSubagentActivityLinesShowsOnlyTraceOpenOperations(t *testing.T) { | |||
| 21 | 21 | "alive": {AgentID: "alive", LastEvent: subagents.TraceEvent{Type: "agent.ready", Timestamp: now.Add(-time.Second)}}, | |
| 22 | 22 | } | |
| 23 | 23 | got := plainActivityLines(renderSubagentActivityLines(tui.Dark, "/", snapshots, views, 80, now)) | |
| 24 | - | want := []string{" / reviewer · tool open · 1m12s"} | |
| 24 | + | want := []string{" / reviewer · running tool · 1m12s"} | |
| 25 | 25 | if strings.Join(got, "\n") != strings.Join(want, "\n") { | |
| 26 | 26 | t.Fatalf("activity lines = %#v, want %#v", got, want) | |
| 27 | 27 | } | |
| 28 | 28 | } | |
| 29 | 29 | ||
| 30 | + | func TestRenderSubagentActivityLinesShowsSafeLastObservation(t *testing.T) { | |
| 31 | + | now := time.Date(2026, time.August, 6, 12, 0, 0, 0, time.UTC) | |
| 32 | + | lines := plainActivityLines(renderSubagentActivityLines(tui.Dark, "/", []subagents.AgentSnapshot{{ID: "review-123", Subagent: "reviewer"}}, map[string]subagents.AgentTraceView{ | |
| 33 | + | "review-123": {OpenOperations: []subagents.Operation{{Type: "turn.started", TurnID: "turn-1", StartedAt: now.Add(-time.Minute)}}, LastObservation: &subagents.LiveObservation{Type: "assistant.stream.observed", TurnID: "turn-1", At: now.Add(-time.Second)}}, | |
| 34 | + | }, 100, now)) | |
| 35 | + | if got := strings.Join(lines, "\n"); !strings.Contains(got, "processing turn · assistant streaming 1s ago") || strings.Contains(got, "delta") { | |
| 36 | + | t.Fatalf("activity lines = %#v", lines) | |
| 37 | + | } | |
| 38 | + | } | |
| 39 | + | ||
| 40 | + | func TestRenderSubagentActivityLinesShowsRecentCompletedToolUnderProviderRequest(t *testing.T) { | |
| 41 | + | now := time.Date(2026, time.August, 6, 12, 0, 0, 0, time.UTC) | |
| 42 | + | lines := plainActivityLines(renderSubagentActivityLines(tui.Dark, "/", []subagents.AgentSnapshot{{ID: "review-123", Subagent: "reviewer"}}, map[string]subagents.AgentTraceView{ | |
| 43 | + | "review-123": {OpenOperations: []subagents.Operation{{Type: "provider.request.started", TurnID: "turn-1", StartedAt: now.Add(-2 * time.Second)}}, LastObservation: &subagents.LiveObservation{Type: "tool.finished", Name: "read", CallID: "call-1", TurnID: "turn-1", At: now.Add(-3 * time.Second)}}, | |
| 44 | + | }, 100, now)) | |
| 45 | + | got := strings.Join(lines, "\n") | |
| 46 | + | if !strings.Contains(got, "zzZz · finished read 3s ago · 2s") { | |
| 47 | + | t.Fatalf("activity lines = %#v", lines) | |
| 48 | + | } | |
| 49 | + | } | |
| 50 | + | ||
| 51 | + | func TestRenderSubagentActivityLineColorsProviderWaitAsActive(t *testing.T) { | |
| 52 | + | now := time.Date(2026, time.August, 6, 12, 0, 0, 0, time.UTC) | |
| 53 | + | operation := subagents.Operation{Type: "provider.request.started", TurnID: "turn-1", StartedAt: now.Add(-2 * time.Second)} | |
| 54 | + | line := renderSubagentActivityLine(tui.Dark, "/", subagents.AgentSnapshot{ID: "review-123", Subagent: "reviewer"}, operation, operation.StartedAt, nil, 100, now) | |
| 55 | + | if !strings.Contains(line, tui.Dark.FGColor(tui.Dark.Spinner, "zzZz")) { | |
| 56 | + | t.Fatalf("provider wait is not rendered with the active color: %q", line) | |
| 57 | + | } | |
| 58 | + | } | |
| 59 | + | ||
| 60 | + | func TestRenderSubagentActivityLineAnimatesProviderWaitAtOneHertz(t *testing.T) { | |
| 61 | + | started := time.Date(2026, time.August, 6, 12, 0, 0, 0, time.UTC) | |
| 62 | + | operation := subagents.Operation{Type: "provider.request.started", TurnID: "turn-1", StartedAt: started} | |
| 63 | + | snapshot := subagents.AgentSnapshot{ID: "review-123", Subagent: "reviewer"} | |
| 64 | + | ||
| 65 | + | wants := []string{"Zzzz", "zZzz", "zzZz", "zzzZ", "Zzzz"} | |
| 66 | + | for second, want := range wants { | |
| 67 | + | line := stripANSIBytes(renderSubagentActivityLine(tui.Dark, "/", snapshot, operation, started, nil, 100, started.Add(time.Duration(second)*time.Second))) | |
| 68 | + | if !strings.Contains(line, "reviewer · "+want+" · ") { | |
| 69 | + | t.Fatalf("provider wait at %ds = %q, want frame %q", second, line, want) | |
| 70 | + | } | |
| 71 | + | } | |
| 72 | + | } | |
| 73 | + | ||
| 74 | + | func TestRenderSubagentActivityLineShowsModelStepBesideElapsedTurnTime(t *testing.T) { | |
| 75 | + | started := time.Date(2026, time.August, 6, 12, 0, 0, 0, time.UTC) | |
| 76 | + | operation := subagents.Operation{Type: "provider.request.started", TurnID: "turn-1", Step: 3, StartedAt: started.Add(8 * time.Second)} | |
| 77 | + | snapshot := subagents.AgentSnapshot{ID: "review-123", Subagent: "reviewer"} | |
| 78 | + | ||
| 79 | + | line := stripANSIBytes(renderSubagentActivityLine(tui.Dark, "/", snapshot, operation, started, nil, 100, started.Add(12*time.Second))) | |
| 80 | + | if !strings.Contains(line, "reviewer · Zzzz · 3 steps · 12s") { | |
| 81 | + | t.Fatalf("provider wait line = %q, want model step beside delegated-turn elapsed time", line) | |
| 82 | + | } | |
| 83 | + | } | |
| 84 | + | ||
| 85 | + | func TestRenderSubagentActivityLinesOmitsPriorTurnObservation(t *testing.T) { | |
| 86 | + | now := time.Date(2026, time.August, 6, 12, 0, 0, 0, time.UTC) | |
| 87 | + | lines := plainActivityLines(renderSubagentActivityLines(tui.Dark, "/", []subagents.AgentSnapshot{{ID: "review-123", Subagent: "reviewer"}}, map[string]subagents.AgentTraceView{ | |
| 88 | + | "review-123": {OpenOperations: []subagents.Operation{{Type: "tool.started", TurnID: "turn-2", StartedAt: now.Add(-time.Minute)}}, LastObservation: &subagents.LiveObservation{Type: "assistant.stream.observed", TurnID: "turn-1", At: now.Add(-time.Second)}}, | |
| 89 | + | }, 100, now)) | |
| 90 | + | if got := strings.Join(lines, "\n"); strings.Contains(got, "assistant streaming") || !strings.Contains(got, "running tool") || !strings.Contains(got, "1m") { | |
| 91 | + | t.Fatalf("activity lines = %#v", lines) | |
| 92 | + | } | |
| 93 | + | } | |
| 94 | + | ||
| 30 | 95 | func TestRenderSubagentActivityLinesKeepsNameAndAgeOnNarrowTerminals(t *testing.T) { | |
| 31 | 96 | now := time.Date(2026, time.August, 6, 12, 0, 0, 0, time.UTC) | |
| 32 | 97 | lines := renderSubagentActivityLines(tui.Dark, "|", []subagents.AgentSnapshot{{ID: "review-123", Subagent: "reviewer"}}, map[string]subagents.AgentTraceView{ | |
| @@ -58,6 +123,19 @@ func TestTruncateSubagentIndicatorTextPreservesGraphemeClusters(t *testing.T) { | |||
| 58 | 123 | } | |
| 59 | 124 | } | |
| 60 | 125 | ||
| 126 | + | func TestRenderSubagentActivityLinesUsesStableDelegatedTurnElapsedTime(t *testing.T) { | |
| 127 | + | now := time.Date(2026, time.August, 6, 12, 0, 0, 0, time.UTC) | |
| 128 | + | turn := subagents.Operation{Type: "turn.started", TurnID: "turn-1", StartedAt: now.Add(-12 * time.Second)} | |
| 129 | + | request := subagents.Operation{Type: "provider.request.started", TurnID: "turn-1", StartedAt: now.Add(-time.Second)} | |
| 130 | + | lines := plainActivityLines(renderSubagentActivityLines(tui.Dark, "/", []subagents.AgentSnapshot{{ID: "review-123", Subagent: "reviewer", Started: now.Add(-5*time.Hour - 16*time.Minute)}}, map[string]subagents.AgentTraceView{ | |
| 131 | + | "review-123": {OpenOperations: []subagents.Operation{request, turn}}, | |
| 132 | + | }, 100, now)) | |
| 133 | + | got := strings.Join(lines, "\n") | |
| 134 | + | if !strings.Contains(got, "zZzz · 12s") || strings.Contains(got, "5h16m") { | |
| 135 | + | t.Fatalf("activity lines = %#v, want current delegated-turn elapsed time", lines) | |
| 136 | + | } | |
| 137 | + | } | |
| 138 | + | ||
| 61 | 139 | func TestFormatSubagentActivityAgeUsesOperationStart(t *testing.T) { | |
| 62 | 140 | now := time.Date(2026, time.August, 6, 12, 0, 0, 0, time.UTC) | |
| 63 | 141 | if got, want := formatSubagentActivityAge(now.Add(-7*time.Second), now), "7s"; got != want { | |
M packages/agent/modes/subagents_dialog.go +27 -18
| @@ -1306,6 +1306,22 @@ func (d *subagentsDialog) renderTranscript(th tui.Theme, width int) []string { | |||
| 1306 | 1306 | " " + th.FGColor(th.Muted, "dir: "+a.Dir), | |
| 1307 | 1307 | " " + th.FGColor(th.Muted, "observation: "+d.traceSummary(a.ID)), | |
| 1308 | 1308 | } | |
| 1309 | + | if request := d.traceView(a.ID).LastRequest; request != nil { | |
| 1310 | + | diagnostic := "provider request: " + request.Outcome | |
| 1311 | + | if request.Provider != "" || request.Model != "" { | |
| 1312 | + | diagnostic += " · " + strings.Trim(strings.TrimSpace(request.Provider+" / "+request.Model), " /") | |
| 1313 | + | } | |
| 1314 | + | if request.Attempt > 0 { | |
| 1315 | + | diagnostic += fmt.Sprintf(" · attempt %d", request.Attempt) | |
| 1316 | + | if request.MaxAttempts > 0 { | |
| 1317 | + | diagnostic += fmt.Sprintf("/%d", request.MaxAttempts) | |
| 1318 | + | } | |
| 1319 | + | } | |
| 1320 | + | if request.ErrorCode != "" { | |
| 1321 | + | diagnostic += " · " + request.ErrorCode | |
| 1322 | + | } | |
| 1323 | + | header = append(header, " "+th.FGColor(th.Muted, diagnostic)) | |
| 1324 | + | } | |
| 1309 | 1325 | if a.Model != "" { | |
| 1310 | 1326 | modelLine := "model: " + a.Model | |
| 1311 | 1327 | if a.Provider != "" { | |
| @@ -1457,17 +1473,7 @@ func (d *subagentsDialog) agentHasOpenOperation(id string) bool { | |||
| 1457 | 1473 | } | |
| 1458 | 1474 | ||
| 1459 | 1475 | func (d *subagentsDialog) traceSummary(id string) string { | |
| 1460 | - | view := d.traceView(id) | |
| 1461 | - | if view.Terminal != "" { | |
| 1462 | - | return view.Terminal | |
| 1463 | - | } | |
| 1464 | - | if len(view.OpenOperations) != 0 { | |
| 1465 | - | return view.OpenOperations[0].Label() | |
| 1466 | - | } | |
| 1467 | - | if view.LastEvent.Type != "" { | |
| 1468 | - | return "last event " + view.LastEvent.Type | |
| 1469 | - | } | |
| 1470 | - | return "no observable operation" | |
| 1476 | + | return d.traceView(id).Summary() | |
| 1471 | 1477 | } | |
| 1472 | 1478 | ||
| 1473 | 1479 | // renderSupervisorTranscriptBlocks converts the agent's flat transcript | |
| @@ -1634,14 +1640,17 @@ func (d *subagentsDialog) traceView(id string) subagents.AgentTraceView { | |||
| 1634 | 1640 | // tool open 3m fix-login-12345 3m inspect main.go | |
| 1635 | 1641 | // completed write-tests-67890 1h add coverage | |
| 1636 | 1642 | func formatSupervisorRow(r subagents.AgentSnapshot, view subagents.AgentTraceView, maxWidth int) string { | |
| 1637 | - | fact := "no observable operation" | |
| 1638 | - | if view.Terminal != "" { | |
| 1639 | - | fact = view.Terminal | |
| 1640 | - | } else if len(view.OpenOperations) != 0 { | |
| 1641 | - | operation := view.OpenOperations[0] | |
| 1642 | - | fact = operation.Label() + " " + formatAge(operation.StartedAt) | |
| 1643 | + | fact := view.Summary() | |
| 1644 | + | if primary, ok := view.Primary(); ok { | |
| 1645 | + | fact = truncateLineSafe(fact, 24) | |
| 1646 | + | if observation := view.ObservationFor(primary); observation != nil { | |
| 1647 | + | fact += " " + formatAge(observation.At) | |
| 1648 | + | } else { | |
| 1649 | + | fact += " " + formatAge(primary.StartedAt) | |
| 1650 | + | } | |
| 1643 | 1651 | } else if view.LastEvent.Type != "" { | |
| 1644 | - | fact = "last " + view.LastEvent.Type + " " + formatAge(view.LastEvent.Timestamp) | |
| 1652 | + | fact = truncateLineSafe(fact, 24) | |
| 1653 | + | fact += " " + formatAge(view.LastEvent.Timestamp) | |
| 1645 | 1654 | } | |
| 1646 | 1655 | age := formatAge(r.Started) | |
| 1647 | 1656 | left := fmt.Sprintf("%-30s %-26s %-8s ", truncateLineSafe(fact, 30), truncateLineSafe(r.ID, 26), age) | |
M packages/agent/modes/subagents_dialog_test.go +50 -3
| @@ -24,6 +24,53 @@ func staticSnapshots(rows ...subagents.AgentSnapshot) func() []subagents.AgentSn | |||
| 24 | 24 | } | |
| 25 | 25 | } | |
| 26 | 26 | ||
| 27 | + | func TestFormatSupervisorRowShowsTerminalResultDelivery(t *testing.T) { | |
| 28 | + | at := time.Date(2026, 8, 12, 7, 0, 0, 0, time.UTC) | |
| 29 | + | got := formatSupervisorRow(subagents.AgentSnapshot{ID: "review-1", Started: at}, subagents.AgentTraceView{ | |
| 30 | + | Terminal: "completed", | |
| 31 | + | Result: &subagents.ResultFact{Available: true, Delivered: true, At: at}, | |
| 32 | + | }, 120) | |
| 33 | + | if !strings.Contains(got, "completed · result delivered") { | |
| 34 | + | t.Fatalf("row = %q", got) | |
| 35 | + | } | |
| 36 | + | } | |
| 37 | + | ||
| 38 | + | func TestFormatSupervisorRowOmitsPriorTurnObservationAge(t *testing.T) { | |
| 39 | + | now := time.Now() | |
| 40 | + | got := formatSupervisorRow(subagents.AgentSnapshot{ID: "review-1", Started: now.Add(-time.Hour)}, subagents.AgentTraceView{ | |
| 41 | + | OpenOperations: []subagents.Operation{{Type: "tool.started", TurnID: "turn-2", StartedAt: now.Add(-time.Minute)}}, | |
| 42 | + | LastObservation: &subagents.LiveObservation{Type: "assistant.stream.observed", TurnID: "turn-1", At: now.Add(-time.Second)}, | |
| 43 | + | }, 120) | |
| 44 | + | if strings.Contains(got, "assistant streaming") || !strings.Contains(got, "running tool") || !strings.Contains(got, "1m") { | |
| 45 | + | t.Fatalf("row = %q", got) | |
| 46 | + | } | |
| 47 | + | } | |
| 48 | + | ||
| 49 | + | func TestFormatSupervisorRowPreservesAgeAfterLongSummary(t *testing.T) { | |
| 50 | + | now := time.Now() | |
| 51 | + | got := formatSupervisorRow(subagents.AgentSnapshot{ID: "review-1", Started: now.Add(-time.Hour)}, subagents.AgentTraceView{ | |
| 52 | + | Terminal: strings.Repeat("completed ", 8), | |
| 53 | + | LastEvent: subagents.TraceEvent{Type: "agent.completed", Timestamp: now.Add(-2 * time.Minute)}, | |
| 54 | + | }, 120) | |
| 55 | + | if !strings.Contains(got, "2m") { | |
| 56 | + | t.Fatalf("row lost fact age: %q", got) | |
| 57 | + | } | |
| 58 | + | } | |
| 59 | + | ||
| 60 | + | func TestSubagentsDialogTranscriptShowsProviderRequestDiagnostic(t *testing.T) { | |
| 61 | + | at := time.Date(2026, 8, 12, 7, 0, 0, 0, time.UTC) | |
| 62 | + | d := newSubagentsDialog() | |
| 63 | + | d.Open(staticSnapshots(subagents.AgentSnapshot{ID: "review-1", Task: "review", Started: at}), nil, nil, nil, nil, nil, "") | |
| 64 | + | d.SetTraceViews(func() map[string]subagents.AgentTraceView { | |
| 65 | + | return map[string]subagents.AgentTraceView{"review-1": {LastRequest: &subagents.RequestFact{Provider: "openai-codex", Model: "gpt-5.6", Attempt: 2, MaxAttempts: 3, Outcome: "failed", ErrorCode: "deadline_exceeded"}}} | |
| 66 | + | }) | |
| 67 | + | d.viewing = true | |
| 68 | + | joined := strings.Join(d.Render(tui.Theme{}, 120), "\n") | |
| 69 | + | if !strings.Contains(joined, "provider request: failed · openai-codex / gpt-5.6 · attempt 2/3 · deadline_exceeded") { | |
| 70 | + | t.Fatalf("diagnostic missing:\n%s", joined) | |
| 71 | + | } | |
| 72 | + | } | |
| 73 | + | ||
| 27 | 74 | func TestSubagentsDialogEmptyState(t *testing.T) { | |
| 28 | 75 | d := newSubagentsDialog() | |
| 29 | 76 | d.Open(staticSnapshots(), nil, nil, nil, nil, nil, "") | |
| @@ -52,7 +99,7 @@ func TestSubagentsDialogRendersRows(t *testing.T) { | |||
| 52 | 99 | }) | |
| 53 | 100 | d.Open(staticSnapshots(rows...), nil, nil, nil, nil, nil, "") | |
| 54 | 101 | out := strings.Join(d.Render(tui.Theme{}, 100), "\n") | |
| 55 | - | for _, want := range []string{"alpha-1", "beta-2", "tool open", "completed", "OPERATION", "TASK"} { | |
| 102 | + | for _, want := range []string{"alpha-1", "beta-2", "running tool", "completed", "OPERATION", "TASK"} { | |
| 56 | 103 | if !strings.Contains(out, want) { | |
| 57 | 104 | t.Fatalf("render missing %q:\n%s", want, out) | |
| 58 | 105 | } | |
| @@ -1314,7 +1361,7 @@ func TestSupervisorTranscriptBusySpinnerRenders(t *testing.T) { | |||
| 1314 | 1361 | d.HandleKey(tui.Key{Kind: tui.KeyEnter}) | |
| 1315 | 1362 | ||
| 1316 | 1363 | out := strings.Join(d.Render(tui.Theme{}, 80), "\n") | |
| 1317 | - | if !strings.Contains(out, "tool open") { | |
| 1364 | + | if !strings.Contains(out, "running tool") { | |
| 1318 | 1365 | t.Fatalf("busy spinner did not surface an open tool near the editor:\n%s", out) | |
| 1319 | 1366 | } | |
| 1320 | 1367 | if d.transcriptSpin == nil { | |
| @@ -1327,7 +1374,7 @@ func TestSupervisorTranscriptBusySpinnerRenders(t *testing.T) { | |||
| 1327 | 1374 | if d.transcriptSpin != nil { | |
| 1328 | 1375 | t.Error("spinner stayed live after agent reported idle") | |
| 1329 | 1376 | } | |
| 1330 | - | if strings.Contains(out, "tool open, ") { | |
| 1377 | + | if strings.Contains(out, "running tool, ") { | |
| 1331 | 1378 | t.Errorf("busy line still rendered after operation finished:\n%s", out) | |
| 1332 | 1379 | } | |
| 1333 | 1380 | } | |
M packages/agent/modes/subagents_slash.go +1 -1
| @@ -261,7 +261,7 @@ func (i *Interactive) runSubagents(ctx context.Context, args []string) { | |||
| 261 | 261 | waitWatcherDone() | |
| 262 | 262 | } | |
| 263 | 263 | }() | |
| 264 | - | if err := a.WaitContext(ctx); err != nil { | |
| 264 | + | if _, err := a.WaitTurnResult(ctx, a.WaitTargetTurnID()); err != nil { | |
| 265 | 265 | return | |
| 266 | 266 | } | |
| 267 | 267 | // If cancellation became observable at the same time as | |
M packages/agent/sdk/sdk.go +1 -4
| @@ -67,7 +67,7 @@ type Config struct { | |||
| 67 | 67 | Temperature *float32 | |
| 68 | 68 | ||
| 69 | 69 | // MaxSteps caps the agent loop iterations per Prompt call. | |
| 70 | - | // 0 uses the default (50). | |
| 70 | + | // 0 leaves the loop unlimited. | |
| 71 | 71 | MaxSteps int | |
| 72 | 72 | ||
| 73 | 73 | // APIKey overrides the credential lookup chain. | |
| @@ -134,9 +134,6 @@ func New(cfg Config) (*Runtime, error) { | |||
| 134 | 134 | WebSearchPolicy: webSearchPolicy, | |
| 135 | 135 | NoSess: true, // SDK callers manage persistence themselves | |
| 136 | 136 | } | |
| 137 | - | if args.MaxSteps == 0 { | |
| 138 | - | args.MaxSteps = 50 | |
| 139 | - | } | |
| 140 | 137 | r, err := agent.Resolve(args, true) | |
| 141 | 138 | if err != nil { | |
| 142 | 139 | return nil, err | |
A packages/agent/sdk/sdk_defaults_test.go +19 -0
| @@ -0,0 +1,19 @@ | |||
| 1 | + | package sdk | |
| 2 | + | ||
| 3 | + | import "testing" | |
| 4 | + | ||
| 5 | + | func TestRuntimeLeavesAgentStepsUnlimitedByDefault(t *testing.T) { | |
| 6 | + | runtime, err := New(Config{ | |
| 7 | + | Provider: "openai", | |
| 8 | + | Model: "gpt-5", | |
| 9 | + | APIKey: "synthetic-test-key", | |
| 10 | + | NoTools: true, | |
| 11 | + | }) | |
| 12 | + | if err != nil { | |
| 13 | + | t.Fatal(err) | |
| 14 | + | } | |
| 15 | + | t.Cleanup(func() { _ = runtime.Close() }) | |
| 16 | + | if runtime.agent.MaxSteps != 0 { | |
| 17 | + | t.Fatalf("max steps = %d, want unlimited", runtime.agent.MaxSteps) | |
| 18 | + | } | |
| 19 | + | } | |
M packages/agent/subagent_policy_test.go +12 -0
| @@ -3,6 +3,7 @@ package agent | |||
| 3 | 3 | import ( | |
| 4 | 4 | "context" | |
| 5 | 5 | "os" | |
| 6 | + | "strings" | |
| 6 | 7 | "testing" | |
| 7 | 8 | "time" | |
| 8 | 9 | ||
| @@ -16,6 +17,17 @@ func TestSubagentPolicyParsesStartupTimeout(t *testing.T) { | |||
| 16 | 17 | } | |
| 17 | 18 | } | |
| 18 | 19 | ||
| 20 | + | func TestLoadConfigRejectsMalformedSubagentDuration(t *testing.T) { | |
| 21 | + | t.Setenv("ZUT_HOME", t.TempDir()) | |
| 22 | + | if err := os.WriteFile(ConfigPath(), []byte(`{"subagents":{"default_timeout":"20"}}`), 0o600); err != nil { | |
| 23 | + | t.Fatal(err) | |
| 24 | + | } | |
| 25 | + | _, err := LoadConfig() | |
| 26 | + | if err == nil || !strings.Contains(err.Error(), "subagents.default_timeout") { | |
| 27 | + | t.Fatalf("LoadConfig error = %v, want field-qualified duration error", err) | |
| 28 | + | } | |
| 29 | + | } | |
| 30 | + | ||
| 19 | 31 | func TestSubagentPolicyIgnoresLegacyMaxTotalSpawned(t *testing.T) { | |
| 20 | 32 | t.Setenv("ZUT_HOME", t.TempDir()) | |
| 21 | 33 | if err := os.WriteFile(ConfigPath(), []byte(`{"subagents":{"max_concurrent":1,"max_total_spawned":1}}`), 0o600); err != nil { | |
M packages/agent/subagent_requirements.go +11 -4
| @@ -3,6 +3,7 @@ package agent | |||
| 3 | 3 | import ( | |
| 4 | 4 | "context" | |
| 5 | 5 | "encoding/json" | |
| 6 | + | "errors" | |
| 6 | 7 | "fmt" | |
| 7 | 8 | "strings" | |
| 8 | 9 | ||
| @@ -65,10 +66,14 @@ func (rt *subagentRuntime) WireRequiredWorkerGate(parent *core.Agent) { | |||
| 65 | 66 | if !allowed { | |
| 66 | 67 | return false, reason, replacement | |
| 67 | 68 | } | |
| 68 | - | rt.acknowledgeSatisfiedRequirements() | |
| 69 | + | if err := rt.acknowledgeSatisfiedRequirements(); err != nil { | |
| 70 | + | return false, err.Error(), "" | |
| 71 | + | } | |
| 69 | 72 | return true, reason, replacement | |
| 70 | 73 | } | |
| 71 | - | rt.acknowledgeSatisfiedRequirements() | |
| 74 | + | if err := rt.acknowledgeSatisfiedRequirements(); err != nil { | |
| 75 | + | return false, err.Error(), "" | |
| 76 | + | } | |
| 72 | 77 | return true, "", "" | |
| 73 | 78 | } | |
| 74 | 79 | } | |
| @@ -105,12 +110,14 @@ func (rt *subagentRuntime) waitRequiredWorkerReady(ctx context.Context) error { | |||
| 105 | 110 | } | |
| 106 | 111 | } | |
| 107 | 112 | ||
| 108 | - | func (rt *subagentRuntime) acknowledgeSatisfiedRequirements() { | |
| 113 | + | func (rt *subagentRuntime) acknowledgeSatisfiedRequirements() error { | |
| 114 | + | var result error | |
| 109 | 115 | for _, snapshot := range rt.supervisor.RequiredSnapshots() { | |
| 110 | 116 | if snapshot.Requirement.State == subagents.RequirementSatisfied && !snapshot.Requirement.Notified { | |
| 111 | - | _ = rt.supervisor.MarkRequirementNotified(snapshot.ID) | |
| 117 | + | result = errors.Join(result, rt.supervisor.MarkRequirementNotified(snapshot.ID)) | |
| 112 | 118 | } | |
| 113 | 119 | } | |
| 120 | + | return result | |
| 114 | 121 | } | |
| 115 | 122 | ||
| 116 | 123 | const requiredWorkerContextRuneBudget = 6000 | |
M packages/agent/subagent_runtime.go +0 -1
| @@ -499,7 +499,6 @@ func subagentPolicyFromConfig(cfg SubagentsConfig) subagents.SubagentPolicy { | |||
| 499 | 499 | QueueTimeout: parseDuration(cfg.QueueTimeout), | |
| 500 | 500 | StartupTimeout: parseDuration(cfg.StartupTimeout), | |
| 501 | 501 | DefaultTimeout: parseDuration(cfg.DefaultTimeout), | |
| 502 | - | MaxTurns: cfg.MaxTurns, | |
| 503 | 502 | MaxOutputBytes: cfg.MaxOutputBytes, | |
| 504 | 503 | MaxOutputLines: cfg.MaxOutputLines, | |
| 505 | 504 | AllowedTools: append([]string(nil), cfg.AllowedTools...), | |
M packages/agent/subagent_runtime_test.go +0 -1
| @@ -59,7 +59,6 @@ func newRuntimeTestConfig(root, repoRoot string, runner func(*subagents.Agent) s | |||
| 59 | 59 | FastMode: true, | |
| 60 | 60 | Policy: subagents.SubagentPolicy{ | |
| 61 | 61 | MaxConcurrent: 2, | |
| 62 | - | MaxTurns: 4, | |
| 63 | 62 | DefaultTimeout: time.Minute, | |
| 64 | 63 | CancelGracePeriod: time.Second, | |
| 65 | 64 | }, | |
M packages/agent/subagent_worker.go +33 -39
| @@ -34,10 +34,9 @@ import ( | |||
| 34 | 34 | // production; tests use the stubchild binary under | |
| 35 | 35 | // packages/agent/subagents/testdata/cmd/stubchild instead of the real model | |
| 36 | 36 | // loop. | |
| 37 | - | // workerTurnBudget is deliberately independent of the model loop: one | |
| 38 | - | // admitted message turn consumes one current-run and lifetime turn, while | |
| 39 | - | // retries, compaction, and provider/tool loops remain inside that turn. | |
| 40 | - | type workerTurnBudget struct { | |
| 37 | + | // workerTurnCounters track stable delegated-turn identity and observability. | |
| 38 | + | // Retries, compaction, and provider/tool loops remain inside one message turn. | |
| 39 | + | type workerTurnCounters struct { | |
| 41 | 40 | sequence int | |
| 42 | 41 | lifetime int | |
| 43 | 42 | current int | |
| @@ -71,17 +70,14 @@ func (s *sessionPersistenceState) err() error { | |||
| 71 | 70 | return s.errValue | |
| 72 | 71 | } | |
| 73 | 72 | ||
| 74 | - | func (b *workerTurnBudget) start(maxTurns int, newRun bool) (step, lifetime, current int, admitted bool) { | |
| 73 | + | func (b *workerTurnCounters) start(newRun bool) (step, lifetime, current int) { | |
| 75 | 74 | if newRun { | |
| 76 | 75 | b.current = 0 | |
| 77 | 76 | } | |
| 78 | 77 | b.sequence++ | |
| 79 | - | if maxTurns > 0 && b.current >= maxTurns { | |
| 80 | - | return b.sequence, b.lifetime, b.current, false | |
| 81 | - | } | |
| 82 | 78 | b.lifetime++ | |
| 83 | 79 | b.current++ | |
| 84 | - | return b.sequence, b.lifetime, b.current, true | |
| 80 | + | return b.sequence, b.lifetime, b.current | |
| 85 | 81 | } | |
| 86 | 82 | ||
| 87 | 83 | func subagentTurnContext(parent context.Context, timeout time.Duration) (context.Context, context.CancelFunc) { | |
| @@ -117,9 +113,10 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) (runE | |||
| 117 | 113 | if err != nil { | |
| 118 | 114 | return err | |
| 119 | 115 | } | |
| 120 | - | extMgr, stopExt := setupNonInteractiveExtensions(ctx, args, &r, version) | |
| 121 | - | defer stopExt() | |
| 122 | - | ||
| 116 | + | // A delegated worker has one bounded task and must not inherit arbitrary | |
| 117 | + | // local extensions, their tools, hooks, or persisted extension state. | |
| 118 | + | // Extensions remain available to explicitly selected skills only through | |
| 119 | + | // the parent process; the worker runs with its resolved built-in registry. | |
| 123 | 120 | ag := r.NewAgent() | |
| 124 | 121 | initialAg := ag | |
| 125 | 122 | defer func() { | |
| @@ -128,7 +125,6 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) (runE | |||
| 128 | 125 | closeAgentLSP(initialAg) | |
| 129 | 126 | } | |
| 130 | 127 | }() | |
| 131 | - | wireNonInteractiveAgentExtHooks(ctx, ag, extMgr) | |
| 132 | 128 | sess, err := openOrCreateSession(ctx, args, r, ag, version) | |
| 133 | 129 | if err != nil { | |
| 134 | 130 | return err | |
| @@ -136,12 +132,11 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) (runE | |||
| 136 | 132 | var sessionPersistence sessionPersistenceState | |
| 137 | 133 | if sess != nil { | |
| 138 | 134 | var providerName, model string | |
| 139 | - | sess, ag, providerName, model, err = applyInitialSessionResume(ctx, args, r, extMgr, sess, ag) | |
| 135 | + | sess, ag, providerName, model, err = applyInitialSessionResume(ctx, args, r, nil, sess, ag) | |
| 140 | 136 | if err != nil { | |
| 141 | 137 | return err | |
| 142 | 138 | } | |
| 143 | 139 | r.Provider, r.Model = providerName, model | |
| 144 | - | ag.CommitToolResult = func(_ string, result core.ToolResult) error { return persistToolResultState(extMgr, sess, result) } | |
| 145 | 140 | // A prompt is acknowledged by the supervisor only after this callback | |
| 146 | 141 | // has appended and synced the matching user message. This closes the | |
| 147 | 142 | // crash window between accepting a follow-up and writing session.json. | |
| @@ -165,8 +160,6 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) (runE | |||
| 165 | 160 | runErr = joinSessionCloseError(runErr, sess) | |
| 166 | 161 | }() | |
| 167 | 162 | } | |
| 168 | - | announceSession(extMgr, sess) | |
| 169 | - | ||
| 170 | 163 | // Resolve retains metadata for catalog models and for valid synthesized | |
| 171 | 164 | // local/routed/open-catalog models. The active agent also survives a session | |
| 172 | 165 | // resume that rebuilt it for a different provider/model pair. | |
| @@ -191,7 +184,6 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) (runE | |||
| 191 | 184 | "model": r.Model, | |
| 192 | 185 | "lifetime_turns": args.SubagentLifetimeTurns, | |
| 193 | 186 | "current_run_turns": args.SubagentRunTurns, | |
| 194 | - | "max_turns": args.SubagentMaxTurns, | |
| 195 | 187 | }) | |
| 196 | 188 | ||
| 197 | 189 | // Keep a per-turn cancel so the "cancel" inbox message can interrupt | |
| @@ -200,7 +192,8 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) (runE | |||
| 200 | 192 | mu sync.Mutex | |
| 201 | 193 | cancelFn context.CancelFunc | |
| 202 | 194 | busyTurn bool | |
| 203 | - | budget = workerTurnBudget{ | |
| 195 | + | counters = workerTurnCounters{ | |
| 196 | + | sequence: args.SubagentLifetimeTurns, | |
| 204 | 197 | lifetime: args.SubagentLifetimeTurns, | |
| 205 | 198 | current: args.SubagentRunTurns, | |
| 206 | 199 | } | |
| @@ -228,26 +221,9 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) (runE | |||
| 228 | 221 | } | |
| 229 | 222 | busyTurn = true | |
| 230 | 223 | turnPending = false | |
| 231 | - | step, lifetime, currentRun, admitted := budget.start(args.SubagentMaxTurns, newRun) | |
| 232 | - | if !admitted { | |
| 233 | - | busyTurn = false | |
| 234 | - | mu.Unlock() | |
| 235 | - | turnID := fmt.Sprintf("turn-%d", step) | |
| 236 | - | errPayload := map[string]any{"code": "turn_rejected", "reason": "max_turns", "command_id": commandID, "message": "maximum subagent turns reached"} | |
| 237 | - | em.emit("error", errPayload) | |
| 238 | - | em.emit("turn.result", map[string]any{"status": "failed", "turn_id": turnID, "error": errPayload}) | |
| 239 | - | em.emit("turn.failed", map[string]any{"turn_id": turnID, "error": errPayload}) | |
| 240 | - | em.emit("turn_end", map[string]any{"step": step, "turn_id": turnID, "error": errPayload["message"]}) | |
| 241 | - | // Max-turn rejection ends the current run, but the worker remains | |
| 242 | - | // alive and can accept a queued follow-up as a fresh run. | |
| 243 | - | em.emit("agent.idle", map[string]any{ | |
| 244 | - | "turn_id": turnID, | |
| 245 | - | "lifetime_turns": lifetime, | |
| 246 | - | "current_run_turns": currentRun, | |
| 247 | - | }) | |
| 248 | - | return | |
| 249 | - | } | |
| 224 | + | step, lifetime, currentRun := counters.start(newRun) | |
| 250 | 225 | turnID := fmt.Sprintf("turn-%d", step) | |
| 226 | + | em.setTurnID(turnID) | |
| 251 | 227 | c, cancel := subagentTurnContext(ctx, args.SubagentTurnTimeout) | |
| 252 | 228 | cancelFn = cancel | |
| 253 | 229 | mu.Unlock() | |
| @@ -354,6 +330,9 @@ func runSubagentWorkerMode(ctx context.Context, args Args, version string) (runE | |||
| 354 | 330 | ||
| 355 | 331 | cancel() | |
| 356 | 332 | mu.Lock() | |
| 333 | + | // Clear the emitter identity before reopening the busy gate: a newly | |
| 334 | + | // admitted turn must not race this completed turn's cleanup. | |
| 335 | + | em.setTurnID("") | |
| 357 | 336 | busyTurn = false | |
| 358 | 337 | cancelFn = nil | |
| 359 | 338 | mu.Unlock() | |
| @@ -487,6 +466,7 @@ type subagentEmitter struct { | |||
| 487 | 466 | mu sync.Mutex | |
| 488 | 467 | w *os.File | |
| 489 | 468 | agentID string | |
| 469 | + | turnID string | |
| 490 | 470 | } | |
| 491 | 471 | ||
| 492 | 472 | func newSubagentEmitter(w *os.File) *subagentEmitter { | |
| @@ -499,6 +479,14 @@ func (e *subagentEmitter) setProtocolIdentity(agentID string) { | |||
| 499 | 479 | e.mu.Unlock() | |
| 500 | 480 | } | |
| 501 | 481 | ||
| 482 | + | // setTurnID associates subsequent worker events with the currently admitted | |
| 483 | + | // delegated turn. Core events do not carry the worker protocol turn ID. | |
| 484 | + | func (e *subagentEmitter) setTurnID(turnID string) { | |
| 485 | + | e.mu.Lock() | |
| 486 | + | e.turnID = turnID | |
| 487 | + | e.mu.Unlock() | |
| 488 | + | } | |
| 489 | + | ||
| 502 | 490 | func (e *subagentEmitter) emit(typ string, data map[string]any) { | |
| 503 | 491 | if data == nil { | |
| 504 | 492 | data = map[string]any{} | |
| @@ -511,6 +499,9 @@ func (e *subagentEmitter) emit(typ string, data map[string]any) { | |||
| 511 | 499 | ||
| 512 | 500 | e.mu.Lock() | |
| 513 | 501 | defer e.mu.Unlock() | |
| 502 | + | if turnID == "" { | |
| 503 | + | turnID = e.turnID | |
| 504 | + | } | |
| 514 | 505 | ||
| 515 | 506 | wireType := canonicalWorkerEvent(typ) | |
| 516 | 507 | envelope := subagents.NewEventEnvelope(wireType, e.agentID, turnID, payload) | |
| @@ -539,7 +530,7 @@ func canonicalWorkerEvent(typ string) string { | |||
| 539 | 530 | return subagents.EventTurnStarted | |
| 540 | 531 | case "turn_progress": | |
| 541 | 532 | return subagents.EventTurnProgress | |
| 542 | - | case "tool_call": | |
| 533 | + | case "tool_execution_started": | |
| 543 | 534 | return subagents.EventToolStarted | |
| 544 | 535 | case "tool_result": | |
| 545 | 536 | return subagents.EventToolFinished | |
| @@ -691,6 +682,9 @@ func resultErrorPayload(err error, shutdownOrigin subagents.ShutdownOrigin) map[ | |||
| 691 | 682 | if provider.IsContextOverflowError(err) { | |
| 692 | 683 | return map[string]any{"code": "context_limit", "message": subagentContextLimitMessage} | |
| 693 | 684 | } | |
| 685 | + | if errors.Is(err, core.ErrStreamIdleTimeout) { | |
| 686 | + | return map[string]any{"code": "stream_idle_timeout", "message": "provider stream produced no events for five minutes; the delegated turn stopped safely"} | |
| 687 | + | } | |
| 694 | 688 | if errors.Is(err, context.DeadlineExceeded) || shutdownOrigin == subagents.ShutdownOriginDeadline { | |
| 695 | 689 | return map[string]any{"code": "deadline_exceeded", "message": "subagent turn deadline exceeded; partial output is preserved in the result and history"} | |
| 696 | 690 | } | |
M packages/agent/subagent_worker_test.go +29 -18
| @@ -52,6 +52,13 @@ func TestSubagentTurnContextResetsDeadlineAndHonorsShutdown(t *testing.T) { | |||
| 52 | 52 | } | |
| 53 | 53 | } | |
| 54 | 54 | ||
| 55 | + | func TestResultErrorPayloadClassifiesStreamIdleTimeout(t *testing.T) { | |
| 56 | + | payload := resultErrorPayload(core.ErrStreamIdleTimeout, "") | |
| 57 | + | if payload["code"] != "stream_idle_timeout" { | |
| 58 | + | t.Fatalf("idle timeout error code = %v, want stream_idle_timeout", payload["code"]) | |
| 59 | + | } | |
| 60 | + | } | |
| 61 | + | ||
| 55 | 62 | func TestResultErrorPayloadClassifiesTurnDeadline(t *testing.T) { | |
| 56 | 63 | payload := resultErrorPayload(context.DeadlineExceeded, "") | |
| 57 | 64 | if payload["code"] != "deadline_exceeded" { | |
| @@ -819,7 +826,8 @@ func TestSupervisorEmitterStdoutShapeMatchesSupervisorParser(t *testing.T) { | |||
| 819 | 826 | defer r.Close() | |
| 820 | 827 | ||
| 821 | 828 | em := newSubagentEmitter(w) | |
| 822 | - | em.emit("turn_start", map[string]any{"step": 1}) | |
| 829 | + | em.setTurnID("turn-1") | |
| 830 | + | em.emit("text_delta", map[string]any{"delta": "sensitive partial answer"}) | |
| 823 | 831 | _ = w.Close() | |
| 824 | 832 | ||
| 825 | 833 | body, err := io.ReadAll(r) | |
| @@ -835,36 +843,39 @@ func TestSupervisorEmitterStdoutShapeMatchesSupervisorParser(t *testing.T) { | |||
| 835 | 843 | if err := json.Unmarshal(lines[0], &object); err != nil { | |
| 836 | 844 | t.Fatalf("not valid json: %v\n%s", err, lines[0]) | |
| 837 | 845 | } | |
| 838 | - | if object["type"] != "turn.started" { | |
| 846 | + | if object["type"] != "message.delta" { | |
| 839 | 847 | t.Errorf("type field missing or wrong: %v", object["type"]) | |
| 840 | 848 | } | |
| 849 | + | if object["turn_id"] != "turn-1" { | |
| 850 | + | t.Errorf("turn_id = %v, want turn-1", object["turn_id"]) | |
| 851 | + | } | |
| 841 | 852 | if _, ok := object["timestamp"].(string); !ok { | |
| 842 | 853 | t.Errorf("timestamp field missing: %v", object["timestamp"]) | |
| 843 | 854 | } | |
| 844 | 855 | payload, ok := object["payload"].(map[string]any) | |
| 845 | - | if !ok || payload["step"] != float64(1) { | |
| 846 | - | t.Errorf("payload step field missing or wrong: %v", object["payload"]) | |
| 856 | + | if !ok || payload["delta"] != "sensitive partial answer" { | |
| 857 | + | t.Errorf("payload delta field missing or wrong: %v", object["payload"]) | |
| 847 | 858 | } | |
| 848 | 859 | } | |
| 849 | 860 | ||
| 850 | - | func TestWorkerTurnBudgetFreshAllowanceOnlyForExplicitRun(t *testing.T) { | |
| 851 | - | var budget workerTurnBudget | |
| 861 | + | func TestWorkerTurnCountersRemainObservableWithoutAdmissionLimit(t *testing.T) { | |
| 862 | + | var counters workerTurnCounters | |
| 852 | 863 | ||
| 853 | - | if _, lifetime, current, admitted := budget.start(2, false); !admitted || lifetime != 1 || current != 1 { | |
| 854 | - | t.Fatalf("first turn = (%d, %d, %v), want (1, 1, true)", lifetime, current, admitted) | |
| 864 | + | if step, lifetime, current := counters.start(false); step != 1 || lifetime != 1 || current != 1 { | |
| 865 | + | t.Fatalf("first turn = (%d, %d, %d), want (1, 1, 1)", step, lifetime, current) | |
| 855 | 866 | } | |
| 856 | - | // Provider retries, model loops, and compaction stay inside the admitted | |
| 857 | - | // turn and therefore do not call start or consume another allowance. | |
| 858 | - | if budget.lifetime != 1 || budget.current != 1 { | |
| 859 | - | t.Fatalf("nested work changed budget to (lifetime=%d, current=%d)", budget.lifetime, budget.current) | |
| 867 | + | // Provider retries, model loops, and compaction stay inside the delegated | |
| 868 | + | // turn and therefore do not advance the message-turn counters. | |
| 869 | + | if counters.lifetime != 1 || counters.current != 1 { | |
| 870 | + | t.Fatalf("nested work changed counters to (lifetime=%d, current=%d)", counters.lifetime, counters.current) | |
| 860 | 871 | } | |
| 861 | - | if _, lifetime, current, admitted := budget.start(2, false); !admitted || lifetime != 2 || current != 2 { | |
| 862 | - | t.Fatalf("second turn = (%d, %d, %v), want (2, 2, true)", lifetime, current, admitted) | |
| 872 | + | if step, lifetime, current := counters.start(false); step != 2 || lifetime != 2 || current != 2 { | |
| 873 | + | t.Fatalf("second turn = (%d, %d, %d), want (2, 2, 2)", step, lifetime, current) | |
| 863 | 874 | } | |
| 864 | - | if _, lifetime, current, admitted := budget.start(2, false); admitted || lifetime != 2 || current != 2 { | |
| 865 | - | t.Fatalf("exhausted turn = (%d, %d, %v), want (2, 2, false)", lifetime, current, admitted) | |
| 875 | + | if step, lifetime, current := counters.start(false); step != 3 || lifetime != 3 || current != 3 { | |
| 876 | + | t.Fatalf("third turn = (%d, %d, %d), want (3, 3, 3)", step, lifetime, current) | |
| 866 | 877 | } | |
| 867 | - | if _, lifetime, current, admitted := budget.start(2, true); !admitted || lifetime != 3 || current != 1 { | |
| 868 | - | t.Fatalf("explicit resumed turn = (%d, %d, %v), want (3, 1, true)", lifetime, current, admitted) | |
| 878 | + | if step, lifetime, current := counters.start(true); step != 4 || lifetime != 4 || current != 1 { | |
| 879 | + | t.Fatalf("explicit resumed turn = (%d, %d, %d), want (4, 4, 1)", step, lifetime, current) | |
| 869 | 880 | } | |
| 870 | 881 | } | |
M packages/agent/subagents/agent.go +86 -18
| @@ -49,7 +49,6 @@ type Agent struct { | |||
| 49 | 49 | WorkspacePath string | |
| 50 | 50 | WorkspaceBase string | |
| 51 | 51 | WorkspaceCapture CaptureMode | |
| 52 | - | MaxTurns int | |
| 53 | 52 | ||
| 54 | 53 | // LifetimeTurns counts every accepted message turn for this worker across | |
| 55 | 54 | // all explicit runs. CurrentRunTurns resets only when an explicit | |
| @@ -58,8 +57,8 @@ type Agent struct { | |||
| 58 | 57 | LifetimeTurns int | |
| 59 | 58 | CurrentRunTurns int | |
| 60 | 59 | ||
| 61 | - | // Timeout is the effective per-agent lifetime, persisted so a resumed | |
| 62 | - | // worker keeps the timeout selected at its spawn boundary. | |
| 60 | + | // Timeout is the effective per-turn deadline. Zero means unlimited. It is | |
| 61 | + | // persisted so a resumed worker keeps the policy selected at spawn time. | |
| 63 | 62 | Timeout time.Duration | |
| 64 | 63 | HeartbeatInterval time.Duration | |
| 65 | 64 | Tools []string | |
| @@ -209,9 +208,11 @@ type Agent struct { | |||
| 209 | 208 | // done closes when the run goroutine finalises the agent's | |
| 210 | 209 | // status (done / failed / killed). Wait blocks on this so | |
| 211 | 210 | // callers don't have to poll. | |
| 212 | - | done chan struct{} | |
| 213 | - | turnResults chan *TurnResult | |
| 214 | - | doneOnce sync.Once | |
| 211 | + | done chan struct{} | |
| 212 | + | results map[string]*TurnResult | |
| 213 | + | resultWaiters map[string]int | |
| 214 | + | resultChanged chan struct{} | |
| 215 | + | doneOnce sync.Once | |
| 215 | 216 | } | |
| 216 | 217 | ||
| 217 | 218 | // Inbox exposes the supervisor-side socket handle. Returns nil for | |
| @@ -270,6 +271,19 @@ func (a *Agent) CurrentTurnID() string { | |||
| 270 | 271 | return a.currentTurnID | |
| 271 | 272 | } | |
| 272 | 273 | ||
| 274 | + | // WaitTargetTurnID returns the delegated turn that a user-facing wait should | |
| 275 | + | // observe. A queued follow-up has not emitted turn.started yet, so derive its | |
| 276 | + | // stable next id from the persisted lifetime counter instead of returning the | |
| 277 | + | // previous turn's completed result. | |
| 278 | + | func (a *Agent) WaitTargetTurnID() string { | |
| 279 | + | a.lifecycleMu.Lock() | |
| 280 | + | defer a.lifecycleMu.Unlock() | |
| 281 | + | if a.turnState == TurnQueued { | |
| 282 | + | return fmt.Sprintf("turn-%d", a.LifetimeTurns+1) | |
| 283 | + | } | |
| 284 | + | return a.currentTurnID | |
| 285 | + | } | |
| 286 | + | ||
| 273 | 287 | // AttemptValue returns the current process attempt number. | |
| 274 | 288 | func (a *Agent) AttemptValue() int { | |
| 275 | 289 | a.lifecycleMu.Lock() | |
| @@ -446,7 +460,17 @@ func (a *Agent) markActivity(now time.Time) { | |||
| 446 | 460 | func (a *Agent) setResult(result *TurnResult) { | |
| 447 | 461 | a.lifecycleMu.Lock() | |
| 448 | 462 | a.result = cloneTurnResult(result) | |
| 463 | + | if result != nil && result.TurnID != "" && a.resultWaiters[result.TurnID] > 0 { | |
| 464 | + | if a.results == nil { | |
| 465 | + | a.results = make(map[string]*TurnResult) | |
| 466 | + | } | |
| 467 | + | a.results[result.TurnID] = cloneTurnResult(result) | |
| 468 | + | } | |
| 449 | 469 | a.updatedAt = time.Now() | |
| 470 | + | if a.resultChanged != nil { | |
| 471 | + | close(a.resultChanged) | |
| 472 | + | } | |
| 473 | + | a.resultChanged = make(chan struct{}) | |
| 450 | 474 | a.lifecycleMu.Unlock() | |
| 451 | 475 | } | |
| 452 | 476 | ||
| @@ -492,8 +516,7 @@ func (a *Agent) Err() error { | |||
| 492 | 516 | return a.lastErr | |
| 493 | 517 | } | |
| 494 | 518 | ||
| 495 | - | // Wait blocks until the agent reaches a terminal state. Used by tests | |
| 496 | - | // and by /subagents wait <id>. | |
| 519 | + | // Wait blocks until the agent process reaches a terminal state. | |
| 497 | 520 | func (a *Agent) Wait() { <-a.done } | |
| 498 | 521 | ||
| 499 | 522 | // WaitContext waits for the agent to finish or for ctx cancellation. | |
| @@ -510,22 +533,67 @@ func (a *Agent) WaitContext(ctx context.Context) error { | |||
| 510 | 533 | } | |
| 511 | 534 | ||
| 512 | 535 | func (a *Agent) waitForTurnResult(ctx context.Context) (*TurnResult, error) { | |
| 536 | + | return a.WaitTurnResult(ctx, a.CurrentTurnID()) | |
| 537 | + | } | |
| 538 | + | ||
| 539 | + | // WaitTurnResult observes one delegated turn without consuming its result or | |
| 540 | + | // affecting the worker when the observer is canceled. An empty turnID accepts | |
| 541 | + | // the next available result. | |
| 542 | + | func (a *Agent) WaitTurnResult(ctx context.Context, turnID string) (*TurnResult, error) { | |
| 513 | 543 | if ctx == nil { | |
| 514 | 544 | ctx = context.Background() | |
| 515 | 545 | } | |
| 516 | - | select { | |
| 517 | - | case result := <-a.turnResults: | |
| 518 | - | if result == nil { | |
| 519 | - | return nil, fmt.Errorf("subagents: nil turn result for %s", a.ID) | |
| 546 | + | if turnID != "" { | |
| 547 | + | a.lifecycleMu.Lock() | |
| 548 | + | if a.resultWaiters == nil { | |
| 549 | + | a.resultWaiters = make(map[string]int) | |
| 520 | 550 | } | |
| 521 | - | return cloneTurnResult(result), nil | |
| 522 | - | case <-a.done: | |
| 523 | - | if result := a.Result(); result != nil { | |
| 551 | + | a.resultWaiters[turnID]++ | |
| 552 | + | a.lifecycleMu.Unlock() | |
| 553 | + | defer func() { | |
| 554 | + | a.lifecycleMu.Lock() | |
| 555 | + | a.resultWaiters[turnID]-- | |
| 556 | + | if a.resultWaiters[turnID] == 0 { | |
| 557 | + | delete(a.resultWaiters, turnID) | |
| 558 | + | delete(a.results, turnID) | |
| 559 | + | } | |
| 560 | + | a.lifecycleMu.Unlock() | |
| 561 | + | }() | |
| 562 | + | } | |
| 563 | + | for { | |
| 564 | + | a.lifecycleMu.Lock() | |
| 565 | + | result := cloneTurnResult(a.result) | |
| 566 | + | if turnID != "" && a.results != nil { | |
| 567 | + | result = cloneTurnResult(a.results[turnID]) | |
| 568 | + | } | |
| 569 | + | if result != nil && (turnID == "" || result.TurnID == turnID) { | |
| 570 | + | a.lifecycleMu.Unlock() | |
| 524 | 571 | return result, nil | |
| 525 | 572 | } | |
| 526 | - | return nil, fmt.Errorf("subagents: agent %s exited without a turn result", a.ID) | |
| 527 | - | case <-ctx.Done(): | |
| 528 | - | return nil, ctx.Err() | |
| 573 | + | if a.resultChanged == nil { | |
| 574 | + | a.resultChanged = make(chan struct{}) | |
| 575 | + | } | |
| 576 | + | changed := a.resultChanged | |
| 577 | + | done := a.done | |
| 578 | + | a.lifecycleMu.Unlock() | |
| 579 | + | ||
| 580 | + | select { | |
| 581 | + | case <-changed: | |
| 582 | + | continue | |
| 583 | + | case <-done: | |
| 584 | + | a.lifecycleMu.Lock() | |
| 585 | + | result := cloneTurnResult(a.result) | |
| 586 | + | if turnID != "" && a.results != nil { | |
| 587 | + | result = cloneTurnResult(a.results[turnID]) | |
| 588 | + | } | |
| 589 | + | a.lifecycleMu.Unlock() | |
| 590 | + | if result != nil && (turnID == "" || result.TurnID == turnID) { | |
| 591 | + | return result, nil | |
| 592 | + | } | |
| 593 | + | return nil, fmt.Errorf("subagents: agent %s exited without turn result %q", a.ID, turnID) | |
| 594 | + | case <-ctx.Done(): | |
| 595 | + | return nil, ctx.Err() | |
| 596 | + | } | |
| 529 | 597 | } | |
| 530 | 598 | } | |
| 531 | 599 | ||
M packages/agent/subagents/agent_test.go +110 -0
| @@ -1,8 +1,11 @@ | |||
| 1 | 1 | package subagents | |
| 2 | 2 | ||
| 3 | 3 | import ( | |
| 4 | + | "context" | |
| 5 | + | "runtime" | |
| 4 | 6 | "sync" | |
| 5 | 7 | "testing" | |
| 8 | + | "time" | |
| 6 | 9 | ) | |
| 7 | 10 | ||
| 8 | 11 | func TestAgentAttemptValueAndTranscriptAccounting(t *testing.T) { | |
| @@ -77,3 +80,110 @@ func TestAgentSnapshotConcurrentTranscriptTruncation(t *testing.T) { | |||
| 77 | 80 | t.Fatal("snapshot did not report transcript truncation") | |
| 78 | 81 | } | |
| 79 | 82 | } | |
| 83 | + | ||
| 84 | + | func TestWaitTurnResultBroadcastsToMultipleObservers(t *testing.T) { | |
| 85 | + | a := &Agent{ID: "worker-1", done: make(chan struct{})} | |
| 86 | + | ctx, cancel := context.WithTimeout(context.Background(), time.Second) | |
| 87 | + | defer cancel() | |
| 88 | + | ||
| 89 | + | results := make(chan *TurnResult, 2) | |
| 90 | + | errs := make(chan error, 2) | |
| 91 | + | for range 2 { | |
| 92 | + | go func() { | |
| 93 | + | result, err := a.WaitTurnResult(ctx, "turn-1") | |
| 94 | + | results <- result | |
| 95 | + | errs <- err | |
| 96 | + | }() | |
| 97 | + | } | |
| 98 | + | a.setResult(&TurnResult{Version: ProtocolVersion, AgentID: a.ID, TurnID: "turn-1", Status: ResultSucceeded}) | |
| 99 | + | ||
| 100 | + | for range 2 { | |
| 101 | + | if err := <-errs; err != nil { | |
| 102 | + | t.Fatalf("wait error = %v", err) | |
| 103 | + | } | |
| 104 | + | if result := <-results; result == nil || result.TurnID != "turn-1" { | |
| 105 | + | t.Fatalf("wait result = %#v, want turn-1", result) | |
| 106 | + | } | |
| 107 | + | } | |
| 108 | + | } | |
| 109 | + | ||
| 110 | + | func TestWaitTurnResultRetainsEarlierCompletedTurn(t *testing.T) { | |
| 111 | + | a := &Agent{ID: "worker-1", done: make(chan struct{})} | |
| 112 | + | resultReady := make(chan *TurnResult, 1) | |
| 113 | + | errReady := make(chan error, 1) | |
| 114 | + | go func() { | |
| 115 | + | result, err := a.WaitTurnResult(context.Background(), "turn-1") | |
| 116 | + | resultReady <- result | |
| 117 | + | errReady <- err | |
| 118 | + | }() | |
| 119 | + | deadline := time.Now().Add(time.Second) | |
| 120 | + | for { | |
| 121 | + | a.lifecycleMu.Lock() | |
| 122 | + | registered := a.resultWaiters["turn-1"] > 0 | |
| 123 | + | a.lifecycleMu.Unlock() | |
| 124 | + | if registered { | |
| 125 | + | break | |
| 126 | + | } | |
| 127 | + | if time.Now().After(deadline) { | |
| 128 | + | t.Fatal("turn-1 waiter did not register") | |
| 129 | + | } | |
| 130 | + | runtime.Gosched() | |
| 131 | + | } | |
| 132 | + | a.setResult(&TurnResult{Version: ProtocolVersion, AgentID: a.ID, TurnID: "turn-1", Status: ResultSucceeded}) | |
| 133 | + | a.setResult(&TurnResult{Version: ProtocolVersion, AgentID: a.ID, TurnID: "turn-2", Status: ResultSucceeded}) | |
| 134 | + | ||
| 135 | + | if err := <-errReady; err != nil { | |
| 136 | + | t.Fatal(err) | |
| 137 | + | } | |
| 138 | + | result := <-resultReady | |
| 139 | + | if result == nil || result.TurnID != "turn-1" { | |
| 140 | + | t.Fatalf("wait result = %#v, want retained turn-1", result) | |
| 141 | + | } | |
| 142 | + | a.lifecycleMu.Lock() | |
| 143 | + | defer a.lifecycleMu.Unlock() | |
| 144 | + | if len(a.results) != 0 || len(a.resultWaiters) != 0 { | |
| 145 | + | t.Fatalf("completed waiter state leaked: results=%d waiters=%d", len(a.results), len(a.resultWaiters)) | |
| 146 | + | } | |
| 147 | + | } | |
| 148 | + | ||
| 149 | + | func TestWaitTurnResultReturnsRequestedResultAfterAgentExit(t *testing.T) { | |
| 150 | + | a := &Agent{ID: "worker-1", done: make(chan struct{}), resultChanged: make(chan struct{})} | |
| 151 | + | resultReady := make(chan *TurnResult, 1) | |
| 152 | + | errReady := make(chan error, 1) | |
| 153 | + | go func() { | |
| 154 | + | result, err := a.WaitTurnResult(context.Background(), "turn-1") | |
| 155 | + | resultReady <- result | |
| 156 | + | errReady <- err | |
| 157 | + | }() | |
| 158 | + | deadline := time.Now().Add(time.Second) | |
| 159 | + | for { | |
| 160 | + | a.lifecycleMu.Lock() | |
| 161 | + | registered := a.resultWaiters["turn-1"] > 0 | |
| 162 | + | a.lifecycleMu.Unlock() | |
| 163 | + | if registered { | |
| 164 | + | break | |
| 165 | + | } | |
| 166 | + | if time.Now().After(deadline) { | |
| 167 | + | t.Fatal("turn-1 waiter did not register") | |
| 168 | + | } | |
| 169 | + | runtime.Gosched() | |
| 170 | + | } | |
| 171 | + | a.lifecycleMu.Lock() | |
| 172 | + | a.results = map[string]*TurnResult{"turn-1": {Version: ProtocolVersion, AgentID: a.ID, TurnID: "turn-1", Status: ResultSucceeded}} | |
| 173 | + | a.result = &TurnResult{Version: ProtocolVersion, AgentID: a.ID, TurnID: "turn-2", Status: ResultSucceeded} | |
| 174 | + | a.lifecycleMu.Unlock() | |
| 175 | + | a.closeDone() | |
| 176 | + | if err := <-errReady; err != nil { | |
| 177 | + | t.Fatal(err) | |
| 178 | + | } | |
| 179 | + | if result := <-resultReady; result == nil || result.TurnID != "turn-1" { | |
| 180 | + | t.Fatalf("wait result = %#v, want retained turn-1", result) | |
| 181 | + | } | |
| 182 | + | } | |
| 183 | + | ||
| 184 | + | func TestWaitTargetTurnIDAdvancesQueuedResume(t *testing.T) { | |
| 185 | + | a := &Agent{LifetimeTurns: 1, turnState: TurnQueued, currentTurnID: "turn-1"} | |
| 186 | + | if got := a.WaitTargetTurnID(); got != "turn-2" { | |
| 187 | + | t.Fatalf("wait target = %q, want turn-2", got) | |
| 188 | + | } | |
| 189 | + | } | |
M packages/agent/subagents/batch.go +29 -8
| @@ -76,6 +76,18 @@ func (b *Batch) Result() *BatchResult { | |||
| 76 | 76 | func (b *Batch) Wait() { <-b.done } | |
| 77 | 77 | func (b *Batch) closeDone() { b.once.Do(func() { close(b.done) }) } | |
| 78 | 78 | ||
| 79 | + | func (b *Batch) WaitContext(ctx context.Context) error { | |
| 80 | + | if ctx == nil { | |
| 81 | + | ctx = context.Background() | |
| 82 | + | } | |
| 83 | + | select { | |
| 84 | + | case <-b.done: | |
| 85 | + | return nil | |
| 86 | + | case <-ctx.Done(): | |
| 87 | + | return ctx.Err() | |
| 88 | + | } | |
| 89 | + | } | |
| 90 | + | ||
| 79 | 91 | func (f *Supervisor) SpawnBatch(ctx context.Context, req BatchRequest) (*Batch, error) { | |
| 80 | 92 | if len(req.Tasks) == 0 { | |
| 81 | 93 | return nil, fmt.Errorf("subagents: batch requires at least one task") | |
| @@ -186,7 +198,7 @@ func (f *Supervisor) collectBatch(batch *Batch) { | |||
| 186 | 198 | } | |
| 187 | 199 | continue | |
| 188 | 200 | } | |
| 189 | - | result, err := f.waitForBatchResult(a) | |
| 201 | + | result, err := f.waitForBatchResult(f.lifetimeCtx, a) | |
| 190 | 202 | if err == nil { | |
| 191 | 203 | results[id] = result | |
| 192 | 204 | switch result.Status { | |
| @@ -222,15 +234,18 @@ func (f *Supervisor) collectBatch(batch *Batch) { | |||
| 222 | 234 | batch.closeDone() | |
| 223 | 235 | } | |
| 224 | 236 | ||
| 225 | - | func (f *Supervisor) waitForBatchResult(a *Agent) (*TurnResult, error) { | |
| 237 | + | func (f *Supervisor) waitForBatchResult(ctx context.Context, a *Agent) (*TurnResult, error) { | |
| 226 | 238 | if a == nil { | |
| 227 | 239 | return nil, errors.New("subagents: missing batch child") | |
| 228 | 240 | } | |
| 241 | + | if ctx == nil { | |
| 242 | + | ctx = context.Background() | |
| 243 | + | } | |
| 229 | 244 | if result := a.Result(); result != nil { | |
| 230 | 245 | return result, nil | |
| 231 | 246 | } | |
| 232 | 247 | if a.Status() != StatusDetached { | |
| 233 | - | return a.waitForTurnResult(context.Background()) | |
| 248 | + | return a.waitForTurnResult(ctx) | |
| 234 | 249 | } | |
| 235 | 250 | ||
| 236 | 251 | // A reloaded worker has a closed supervisor-side done channel even when | |
| @@ -268,8 +283,6 @@ func (f *Supervisor) waitForBatchResult(a *Agent) (*TurnResult, error) { | |||
| 268 | 283 | } | |
| 269 | 284 | ||
| 270 | 285 | interval := 100 * time.Millisecond | |
| 271 | - | deadline := time.NewTimer(f.cfg.Policy.DefaultTimeout) | |
| 272 | - | defer deadline.Stop() | |
| 273 | 286 | ticker := time.NewTicker(interval) | |
| 274 | 287 | defer ticker.Stop() | |
| 275 | 288 | for { | |
| @@ -281,20 +294,28 @@ func (f *Supervisor) waitForBatchResult(a *Agent) (*TurnResult, error) { | |||
| 281 | 294 | if !inboxLive(a.InboxPath) { | |
| 282 | 295 | return nil, fmt.Errorf("subagents: agent %s exited without a turn result", a.ID) | |
| 283 | 296 | } | |
| 284 | - | case <-deadline.C: | |
| 285 | - | return nil, fmt.Errorf("subagents: agent %s batch result timed out", a.ID) | |
| 297 | + | case <-ctx.Done(): | |
| 298 | + | return nil, ctx.Err() | |
| 286 | 299 | } | |
| 287 | 300 | } | |
| 288 | 301 | } | |
| 289 | 302 | ||
| 290 | 303 | func (f *Supervisor) WaitBatch(id string) (*BatchResult, error) { | |
| 304 | + | return f.WaitBatchContext(context.Background(), id) | |
| 305 | + | } | |
| 306 | + | ||
| 307 | + | // WaitBatchContext waits for the aggregate result without changing the batch | |
| 308 | + | // or any child when the observer's context ends. | |
| 309 | + | func (f *Supervisor) WaitBatchContext(ctx context.Context, id string) (*BatchResult, error) { | |
| 291 | 310 | f.mu.Lock() | |
| 292 | 311 | batch := f.batches[id] | |
| 293 | 312 | f.mu.Unlock() | |
| 294 | 313 | if batch == nil { | |
| 295 | 314 | return nil, fmt.Errorf("subagents: no such batch %q", id) | |
| 296 | 315 | } | |
| 297 | - | batch.Wait() | |
| 316 | + | if err := batch.WaitContext(ctx); err != nil { | |
| 317 | + | return nil, err | |
| 318 | + | } | |
| 298 | 319 | return batch.Result(), nil | |
| 299 | 320 | } | |
| 300 | 321 | ||
M packages/agent/subagents/batch_test.go +44 -0
| @@ -3,6 +3,7 @@ package subagents | |||
| 3 | 3 | import ( | |
| 4 | 4 | "context" | |
| 5 | 5 | "encoding/json" | |
| 6 | + | "errors" | |
| 6 | 7 | "fmt" | |
| 7 | 8 | "os" | |
| 8 | 9 | "path/filepath" | |
| @@ -137,6 +138,49 @@ func TestBatchSpawnWaitAndCollect(t *testing.T) { | |||
| 137 | 138 | } | |
| 138 | 139 | } | |
| 139 | 140 | ||
| 141 | + | func TestBatchResultWaitCancellationDoesNotCancelWorker(t *testing.T) { | |
| 142 | + | started := make(chan struct{}) | |
| 143 | + | release := make(chan struct{}) | |
| 144 | + | f := New(Config{ | |
| 145 | + | Root: t.TempDir(), RepoRoot: t.TempDir(), | |
| 146 | + | NewRunner: func(*Agent) Runner { | |
| 147 | + | return RunnerFunc(func(context.Context, Sink) error { | |
| 148 | + | close(started) | |
| 149 | + | <-release | |
| 150 | + | return nil | |
| 151 | + | }) | |
| 152 | + | }, | |
| 153 | + | }) | |
| 154 | + | t.Cleanup(f.StopAll) | |
| 155 | + | batch, err := f.SpawnBatch(context.Background(), BatchRequest{Tasks: []string{"one"}}) | |
| 156 | + | if err != nil { | |
| 157 | + | t.Fatal(err) | |
| 158 | + | } | |
| 159 | + | <-started | |
| 160 | + | ctx, cancel := context.WithCancel(context.Background()) | |
| 161 | + | cancel() | |
| 162 | + | ||
| 163 | + | result, err := f.WaitBatchContext(ctx, batch.ID) | |
| 164 | + | if !errors.Is(err, context.Canceled) { | |
| 165 | + | t.Fatalf("wait error = %v, want context.Canceled", err) | |
| 166 | + | } | |
| 167 | + | if result != nil { | |
| 168 | + | t.Fatalf("wait result = %#v, want nil", result) | |
| 169 | + | } | |
| 170 | + | child := f.Get(batch.ChildIDs[0]) | |
| 171 | + | if child == nil || child.Status() != StatusRunning { | |
| 172 | + | t.Fatalf("worker after observer cancellation = %#v, want running", child) | |
| 173 | + | } | |
| 174 | + | close(release) | |
| 175 | + | completed, err := f.WaitBatch(batch.ID) | |
| 176 | + | if err != nil { | |
| 177 | + | t.Fatal(err) | |
| 178 | + | } | |
| 179 | + | if completed.Status != BatchSucceeded || completed.Results[child.ID] == nil { | |
| 180 | + | t.Fatalf("completed batch = %#v, want successful retained worker result", completed) | |
| 181 | + | } | |
| 182 | + | } | |
| 183 | + | ||
| 140 | 184 | func TestBatchAllowsMoreThanFormerLifetimeLimit(t *testing.T) { | |
| 141 | 185 | const workersBeyondFormerLifetimeLimit = 33 | |
| 142 | 186 | tasks := make([]string, workersBeyondFormerLifetimeLimit) | |
M packages/agent/subagents/model.go +7 -10
| @@ -106,16 +106,16 @@ const ( | |||
| 106 | 106 | ) | |
| 107 | 107 | ||
| 108 | 108 | // SubagentPolicy contains manager-owned safety and resource limits. normalize | |
| 109 | - | // replaces zero MaxOutputBytes, MaxOutputLines, QueueTimeout, StartupTimeout, | |
| 110 | - | // and DefaultTimeout values with positive manager defaults. Other zero values | |
| 111 | - | // retain their documented semantics. | |
| 109 | + | // replaces zero MaxOutputBytes, MaxOutputLines, QueueTimeout, and StartupTimeout | |
| 110 | + | // values with positive manager defaults. Zero DefaultTimeout leaves delegated | |
| 111 | + | // turns unlimited; queue, startup, stream-idle, and idle-process limits remain | |
| 112 | + | // independently bounded. | |
| 112 | 113 | type SubagentPolicy struct { | |
| 113 | 114 | MaxConcurrent int | |
| 114 | 115 | MaxConcurrentPerParent int | |
| 115 | 116 | QueueTimeout time.Duration | |
| 116 | 117 | StartupTimeout time.Duration | |
| 117 | 118 | DefaultTimeout time.Duration | |
| 118 | - | MaxTurns int | |
| 119 | 119 | MaxOutputBytes int | |
| 120 | 120 | MaxOutputLines int | |
| 121 | 121 | AllowedTools []string | |
| @@ -127,6 +127,9 @@ type SubagentPolicy struct { | |||
| 127 | 127 | } | |
| 128 | 128 | ||
| 129 | 129 | func (p *SubagentPolicy) normalize() { | |
| 130 | + | if p.DefaultTimeout < 0 { | |
| 131 | + | p.DefaultTimeout = 0 | |
| 132 | + | } | |
| 130 | 133 | if p.MaxConcurrent <= 0 { | |
| 131 | 134 | p.MaxConcurrent = 8 | |
| 132 | 135 | } | |
| @@ -139,12 +142,6 @@ func (p *SubagentPolicy) normalize() { | |||
| 139 | 142 | if p.StartupTimeout <= 0 { | |
| 140 | 143 | p.StartupTimeout = time.Minute | |
| 141 | 144 | } | |
| 142 | - | if p.DefaultTimeout <= 0 { | |
| 143 | - | p.DefaultTimeout = 20 * time.Minute | |
| 144 | - | } | |
| 145 | - | if p.MaxTurns <= 0 { | |
| 146 | - | p.MaxTurns = 3 | |
| 147 | - | } | |
| 148 | 145 | if p.MaxOutputBytes <= 0 { | |
| 149 | 146 | p.MaxOutputBytes = 500_000 | |
| 150 | 147 | } | |
M packages/agent/subagents/persist.go +42 -17
| @@ -57,10 +57,10 @@ type agentMeta struct { | |||
| 57 | 57 | WorkspacePath string `json:"workspace_path,omitempty"` | |
| 58 | 58 | WorkspaceBase string `json:"workspace_base,omitempty"` | |
| 59 | 59 | WorkspaceCapture CaptureMode `json:"workspace_capture,omitempty"` | |
| 60 | - | MaxTurns int `json:"max_turns,omitempty"` | |
| 61 | 60 | LifetimeTurns int `json:"lifetime_turns,omitempty"` | |
| 62 | 61 | CurrentRunTurns int `json:"current_run_turns,omitempty"` | |
| 63 | 62 | Timeout time.Duration `json:"timeout,omitempty"` | |
| 63 | + | TurnTimeoutMode turnTimeoutMode `json:"turn_timeout_mode,omitempty"` | |
| 64 | 64 | Tools []string `json:"tools,omitempty"` | |
| 65 | 65 | WebSearchPolicy WebSearchPolicy `json:"web_search_policy,omitempty"` | |
| 66 | 66 | Status Status `json:"status,omitempty"` | |
| @@ -96,6 +96,36 @@ type agentMeta struct { | |||
| 96 | 96 | SessionID string `json:"session_id,omitempty"` | |
| 97 | 97 | } | |
| 98 | 98 | ||
| 99 | + | type turnTimeoutMode string | |
| 100 | + | ||
| 101 | + | const ( | |
| 102 | + | turnTimeoutUnlimited turnTimeoutMode = "unlimited" | |
| 103 | + | turnTimeoutLimited turnTimeoutMode = "limited" | |
| 104 | + | ) | |
| 105 | + | ||
| 106 | + | func timeoutModeFor(timeout time.Duration) turnTimeoutMode { | |
| 107 | + | if timeout > 0 { | |
| 108 | + | return turnTimeoutLimited | |
| 109 | + | } | |
| 110 | + | return turnTimeoutUnlimited | |
| 111 | + | } | |
| 112 | + | ||
| 113 | + | func resolvePersistedTurnTimeout(m agentMeta, fallback time.Duration) time.Duration { | |
| 114 | + | switch m.TurnTimeoutMode { | |
| 115 | + | case turnTimeoutUnlimited: | |
| 116 | + | return 0 | |
| 117 | + | case turnTimeoutLimited: | |
| 118 | + | return m.Timeout | |
| 119 | + | default: | |
| 120 | + | // Legacy metadata stored the effective positive timeout. A missing | |
| 121 | + | // value inherited the then-current supervisor policy. | |
| 122 | + | if m.Timeout > 0 { | |
| 123 | + | return m.Timeout | |
| 124 | + | } | |
| 125 | + | return fallback | |
| 126 | + | } | |
| 127 | + | } | |
| 128 | + | ||
| 99 | 129 | func metaPath(stateDir string) string { return filepath.Join(stateDir, "meta.json") } | |
| 100 | 130 | ||
| 101 | 131 | // writeAgentMeta serialises a's identity into stateDir/meta.json. The | |
| @@ -159,10 +189,10 @@ func writeAgentMeta(stateDir string, a *Agent) error { | |||
| 159 | 189 | WorkspacePath: a.WorkspacePath, | |
| 160 | 190 | WorkspaceBase: a.WorkspaceBase, | |
| 161 | 191 | WorkspaceCapture: a.WorkspaceCapture, | |
| 162 | - | MaxTurns: a.MaxTurns, | |
| 163 | 192 | LifetimeTurns: lifetimeTurns, | |
| 164 | 193 | CurrentRunTurns: currentRunTurns, | |
| 165 | 194 | Timeout: a.Timeout, | |
| 195 | + | TurnTimeoutMode: timeoutModeFor(a.Timeout), | |
| 166 | 196 | Tools: append([]string(nil), a.Tools...), | |
| 167 | 197 | WebSearchPolicy: webSearchPolicy, | |
| 168 | 198 | Status: status, | |
| @@ -251,6 +281,12 @@ func readAgentMeta(stateDir string) (agentMeta, error) { | |||
| 251 | 281 | if m.ID == "" { | |
| 252 | 282 | return m, fmt.Errorf("subagent meta %s: missing id", stateDir) | |
| 253 | 283 | } | |
| 284 | + | if m.TurnTimeoutMode != "" && m.TurnTimeoutMode != turnTimeoutUnlimited && m.TurnTimeoutMode != turnTimeoutLimited { | |
| 285 | + | return m, fmt.Errorf("subagent meta %s: invalid turn_timeout_mode %q", stateDir, m.TurnTimeoutMode) | |
| 286 | + | } | |
| 287 | + | if m.TurnTimeoutMode == turnTimeoutLimited && m.Timeout <= 0 { | |
| 288 | + | return m, fmt.Errorf("subagent meta %s: limited turn timeout must be positive", stateDir) | |
| 289 | + | } | |
| 254 | 290 | return m, nil | |
| 255 | 291 | } | |
| 256 | 292 | ||
| @@ -495,10 +531,9 @@ func (f *Supervisor) buildDetachedAgent(m agentMeta) (*Agent, bool) { | |||
| 495 | 531 | WorkspacePath: m.WorkspacePath, | |
| 496 | 532 | WorkspaceBase: m.WorkspaceBase, | |
| 497 | 533 | WorkspaceCapture: m.WorkspaceCapture, | |
| 498 | - | MaxTurns: m.MaxTurns, | |
| 499 | 534 | LifetimeTurns: m.LifetimeTurns, | |
| 500 | 535 | CurrentRunTurns: m.CurrentRunTurns, | |
| 501 | - | Timeout: m.Timeout, | |
| 536 | + | Timeout: resolvePersistedTurnTimeout(m, f.cfg.Policy.DefaultTimeout), | |
| 502 | 537 | Tools: append([]string(nil), m.Tools...), | |
| 503 | 538 | WebSearchPolicy: childWebSearchPolicy(m.WebSearchPolicy, m.Subagent, m.Tools), | |
| 504 | 539 | Attempt: m.Attempt, | |
| @@ -530,7 +565,6 @@ func (f *Supervisor) buildDetachedAgent(m agentMeta) (*Agent, bool) { | |||
| 530 | 565 | maxOutputBytes: f.cfg.Policy.MaxOutputBytes, | |
| 531 | 566 | maxOutputLines: f.cfg.Policy.MaxOutputLines, | |
| 532 | 567 | done: make(chan struct{}), | |
| 533 | - | turnResults: make(chan *TurnResult, 16), | |
| 534 | 568 | persistFn: f.persistAgent, | |
| 535 | 569 | } | |
| 536 | 570 | if a.updatedAt.IsZero() { | |
| @@ -1193,7 +1227,7 @@ func (f *Supervisor) resumeWithHook(ctx context.Context, id string, resuming boo | |||
| 1193 | 1227 | Subagent: existing.Subagent, | |
| 1194 | 1228 | WorkspaceMode: existing.WorkspaceMode, WorkspacePath: existing.WorkspacePath, | |
| 1195 | 1229 | WorkspaceBase: existing.WorkspaceBase, WorkspaceCapture: existing.WorkspaceCapture, | |
| 1196 | - | MaxTurns: existing.MaxTurns, LifetimeTurns: lifetimeTurns, CurrentRunTurns: currentRunTurns, Timeout: existing.Timeout, Tools: append([]string(nil), existing.Tools...), | |
| 1230 | + | LifetimeTurns: lifetimeTurns, CurrentRunTurns: currentRunTurns, Timeout: existing.Timeout, TurnTimeoutMode: timeoutModeFor(existing.Timeout), Tools: append([]string(nil), existing.Tools...), | |
| 1197 | 1231 | WebSearchPolicy: childWebSearchPolicy(existing.WebSearchPolicy, existing.Subagent, existing.Tools), | |
| 1198 | 1232 | CurrentTurnID: existingSnapshot.CurrentTurnID, Attempt: existingSnapshot.Attempt, | |
| 1199 | 1233 | Requirement: existingSnapshot.Requirement, | |
| @@ -1209,7 +1243,7 @@ func (f *Supervisor) resumeWithHook(ctx context.Context, id string, resuming boo | |||
| 1209 | 1243 | if len(m.Tools) == 0 && len(f.cfg.Policy.AllowedTools) > 0 { | |
| 1210 | 1244 | m.Tools = append([]string(nil), f.cfg.Policy.AllowedTools...) | |
| 1211 | 1245 | } | |
| 1212 | - | if err := f.validateSpawnOptions(SpawnRequest{MaxTurns: m.MaxTurns, Tools: m.Tools}); err != nil { | |
| 1246 | + | if err := f.validateSpawnOptions(SpawnRequest{Tools: m.Tools}); err != nil { | |
| 1213 | 1247 | return nil, err | |
| 1214 | 1248 | } | |
| 1215 | 1249 | workspaceMode := m.WorkspaceMode | |
| @@ -1229,10 +1263,6 @@ func (f *Supervisor) resumeWithHook(ctx context.Context, id string, resuming boo | |||
| 1229 | 1263 | if err := validateWorkspaceRoot(repositoryRoot, workspaceMode, allowedRoots); err != nil { | |
| 1230 | 1264 | return nil, err | |
| 1231 | 1265 | } | |
| 1232 | - | maxTurns := m.MaxTurns | |
| 1233 | - | if maxTurns <= 0 { | |
| 1234 | - | maxTurns = f.cfg.Policy.MaxTurns | |
| 1235 | - | } | |
| 1236 | 1266 | existingWorkspacePath := "" | |
| 1237 | 1267 | if workspaceMode == WorkspaceWorktree { | |
| 1238 | 1268 | existingWorkspacePath = m.WorkspacePath | |
| @@ -1254,10 +1284,7 @@ func (f *Supervisor) resumeWithHook(ctx context.Context, id string, resuming boo | |||
| 1254 | 1284 | m.RepositoryRoot = workspace.RepositoryRoot() | |
| 1255 | 1285 | m.WorkspacePath = workspace.Dir() | |
| 1256 | 1286 | m.WorkspaceMode = workspace.Mode() | |
| 1257 | - | timeout := m.Timeout | |
| 1258 | - | if timeout <= 0 { | |
| 1259 | - | timeout = f.cfg.Policy.DefaultTimeout | |
| 1260 | - | } | |
| 1287 | + | timeout := resolvePersistedTurnTimeout(m, f.cfg.Policy.DefaultTimeout) | |
| 1261 | 1288 | runCtx, cancel := f.workerContext() | |
| 1262 | 1289 | a := &Agent{ | |
| 1263 | 1290 | ID: m.ID, | |
| @@ -1281,7 +1308,6 @@ func (f *Supervisor) resumeWithHook(ctx context.Context, id string, resuming boo | |||
| 1281 | 1308 | WorkspacePath: m.WorkspacePath, | |
| 1282 | 1309 | WorkspaceBase: m.WorkspaceBase, | |
| 1283 | 1310 | WorkspaceCapture: m.WorkspaceCapture, | |
| 1284 | - | MaxTurns: maxTurns, | |
| 1285 | 1311 | LifetimeTurns: m.LifetimeTurns, | |
| 1286 | 1312 | CurrentRunTurns: m.CurrentRunTurns, | |
| 1287 | 1313 | Timeout: timeout, | |
| @@ -1312,7 +1338,6 @@ func (f *Supervisor) resumeWithHook(ctx context.Context, id string, resuming boo | |||
| 1312 | 1338 | maxOutputBytes: f.cfg.Policy.MaxOutputBytes, | |
| 1313 | 1339 | maxOutputLines: f.cfg.Policy.MaxOutputLines, | |
| 1314 | 1340 | done: make(chan struct{}), | |
| 1315 | - | turnResults: make(chan *TurnResult, 16), | |
| 1316 | 1341 | } | |
| 1317 | 1342 | // Carry the previous transcript forward so the dashboard doesn't | |
| 1318 | 1343 | // flash empty between resume and the first new event. | |
M packages/agent/subagents/persist_test.go +49 -0
| @@ -218,6 +218,55 @@ func TestResumePreservesPerAgentTimeout(t *testing.T) { | |||
| 218 | 218 | resumed.Wait() | |
| 219 | 219 | } | |
| 220 | 220 | ||
| 221 | + | func TestResumePreservesUnlimitedTurnTimeout(t *testing.T) { | |
| 222 | + | root := t.TempDir() | |
| 223 | + | first := New(Config{ | |
| 224 | + | Root: root, RepoRoot: root, | |
| 225 | + | NewRunner: func(*Agent) Runner { | |
| 226 | + | return RunnerFunc(func(ctx context.Context, _ Sink) error { | |
| 227 | + | <-ctx.Done() | |
| 228 | + | return ctx.Err() | |
| 229 | + | }) | |
| 230 | + | }, | |
| 231 | + | }) | |
| 232 | + | a, err := first.Spawn(context.Background(), "unlimited timeout test") | |
| 233 | + | if err != nil { | |
| 234 | + | t.Fatal(err) | |
| 235 | + | } | |
| 236 | + | first.StopAll() | |
| 237 | + | a.Wait() | |
| 238 | + | ||
| 239 | + | observedCh := make(chan time.Duration, 1) | |
| 240 | + | second := New(Config{ | |
| 241 | + | Root: root, RepoRoot: root, | |
| 242 | + | Policy: SubagentPolicy{DefaultTimeout: time.Hour}, | |
| 243 | + | NewRunner: func(a *Agent) Runner { | |
| 244 | + | observedCh <- a.Timeout | |
| 245 | + | return RunnerFunc(func(ctx context.Context, _ Sink) error { | |
| 246 | + | <-ctx.Done() | |
| 247 | + | return ctx.Err() | |
| 248 | + | }) | |
| 249 | + | }, | |
| 250 | + | }) | |
| 251 | + | if loaded, errs := second.Reload(); loaded != 1 || len(errs) != 0 { | |
| 252 | + | t.Fatalf("reload loaded=%d errs=%v", loaded, errs) | |
| 253 | + | } | |
| 254 | + | resumed, err := second.ResumeSession(context.Background(), a.ID) | |
| 255 | + | if err != nil { | |
| 256 | + | t.Fatal(err) | |
| 257 | + | } | |
| 258 | + | select { | |
| 259 | + | case observed := <-observedCh: | |
| 260 | + | if resumed.Timeout != 0 || observed != 0 { | |
| 261 | + | t.Fatalf("resumed timeout = %s, runner observed %s; want unlimited", resumed.Timeout, observed) | |
| 262 | + | } | |
| 263 | + | case <-time.After(time.Second): | |
| 264 | + | t.Fatal("timed out waiting for resumed runner") | |
| 265 | + | } | |
| 266 | + | second.StopAll() | |
| 267 | + | resumed.Wait() | |
| 268 | + | } | |
| 269 | + | ||
| 221 | 270 | func TestReloadRejectsMetadataPathsOutsideState(t *testing.T) { | |
| 222 | 271 | root := t.TempDir() | |
| 223 | 272 | outside := filepath.Join(t.TempDir(), "outside.json") | |
M packages/agent/subagents/projection.go +375 -33
| @@ -10,11 +10,17 @@ import ( | |||
| 10 | 10 | // boundary. Completed operations are deliberately not retained in the public | |
| 11 | 11 | // projection: their terminal trace fact remains available as LastEvent. | |
| 12 | 12 | type Operation struct { | |
| 13 | - | Type string | |
| 14 | - | AgentID string | |
| 15 | - | TurnID string | |
| 16 | - | CallID string | |
| 17 | - | StartedAt time.Time | |
| 13 | + | Type string | |
| 14 | + | AgentID string | |
| 15 | + | TurnID string | |
| 16 | + | CallID string | |
| 17 | + | Name string | |
| 18 | + | Provider string | |
| 19 | + | Model string | |
| 20 | + | Step int | |
| 21 | + | Attempt int | |
| 22 | + | MaxAttempts int | |
| 23 | + | StartedAt time.Time | |
| 18 | 24 | } | |
| 19 | 25 | ||
| 20 | 26 | // Open is always true because the projection exposes only unmatched starts. | |
| @@ -29,55 +35,229 @@ func (o Operation) Duration(now time.Time) time.Duration { | |||
| 29 | 35 | return now.Sub(o.StartedAt) | |
| 30 | 36 | } | |
| 31 | 37 | ||
| 32 | - | // Label returns the shared concise label for an open operation. | |
| 33 | - | func (o Operation) Label() string { return strings.TrimSuffix(o.Type, ".started") + " open" } | |
| 38 | + | // Label returns a concise user-facing description of an open operation. | |
| 39 | + | func (o Operation) Label() string { | |
| 40 | + | switch o.Type { | |
| 41 | + | case "provider.request.started": | |
| 42 | + | return "waiting for model response" | |
| 43 | + | case "tool.started": | |
| 44 | + | if o.Name != "" { | |
| 45 | + | return "running " + o.Name | |
| 46 | + | } | |
| 47 | + | return "running tool" | |
| 48 | + | case "turn.started": | |
| 49 | + | return "processing turn" | |
| 50 | + | case "agent.wait.started": | |
| 51 | + | return "waiting" | |
| 52 | + | default: | |
| 53 | + | return strings.ReplaceAll(strings.TrimSuffix(o.Type, ".started"), ".", " ") | |
| 54 | + | } | |
| 55 | + | } | |
| 56 | + | ||
| 57 | + | // LiveObservation is the last safe, user-facing signal received from a | |
| 58 | + | // worker. It intentionally contains no worker text, tool arguments, output, | |
| 59 | + | // or low-level protocol event names. | |
| 60 | + | type LiveObservation struct { | |
| 61 | + | Type string | |
| 62 | + | Source string | |
| 63 | + | At time.Time | |
| 64 | + | TurnID string | |
| 65 | + | CallID string | |
| 66 | + | Name string | |
| 67 | + | } | |
| 68 | + | ||
| 69 | + | // Label returns a concise safe description of this observation. | |
| 70 | + | func (o LiveObservation) Label() string { | |
| 71 | + | switch o.Type { | |
| 72 | + | case "assistant.stream.observed": | |
| 73 | + | return "assistant streaming" | |
| 74 | + | case "reasoning.stream.observed": | |
| 75 | + | return "reasoning stream observed" | |
| 76 | + | case "tool.output.observed": | |
| 77 | + | if o.Name != "" { | |
| 78 | + | return "output from " + o.Name | |
| 79 | + | } | |
| 80 | + | return "tool output" | |
| 81 | + | case "tool.finished": | |
| 82 | + | if o.Name != "" { | |
| 83 | + | return "finished " + o.Name | |
| 84 | + | } | |
| 85 | + | return "tool finished" | |
| 86 | + | default: | |
| 87 | + | return strings.ReplaceAll(o.Type, ".", " ") | |
| 88 | + | } | |
| 89 | + | } | |
| 90 | + | ||
| 91 | + | // ResultFact describes a durable result and whether the parent accepted its | |
| 92 | + | // required-work notification. Available is not equivalent to Delivered. | |
| 93 | + | type ResultFact struct { | |
| 94 | + | Available bool | |
| 95 | + | Ref string | |
| 96 | + | Delivered bool | |
| 97 | + | Failed bool | |
| 98 | + | At time.Time | |
| 99 | + | } | |
| 100 | + | ||
| 101 | + | // RequestFact is a bounded provider-attempt diagnostic. It excludes request | |
| 102 | + | // bodies, streamed text, tool arguments, and raw provider errors. | |
| 103 | + | type RequestFact struct { | |
| 104 | + | TurnID string | |
| 105 | + | Provider string | |
| 106 | + | Model string | |
| 107 | + | Attempt int | |
| 108 | + | MaxAttempts int | |
| 109 | + | Outcome string | |
| 110 | + | ErrorCode string | |
| 111 | + | StartedAt time.Time | |
| 112 | + | EndedAt time.Time | |
| 113 | + | } | |
| 34 | 114 | ||
| 35 | 115 | // AgentTraceView is the only user-facing execution state. It intentionally | |
| 36 | 116 | // contains operations and terminal facts rather than lifecycle guesses such as | |
| 37 | 117 | // working, idle, or alive. | |
| 38 | 118 | type AgentTraceView struct { | |
| 39 | - | AgentID string | |
| 40 | - | LastEvent TraceEvent | |
| 41 | - | OpenOperations []Operation | |
| 42 | - | Terminal string | |
| 119 | + | AgentID string | |
| 120 | + | LastEvent TraceEvent | |
| 121 | + | LastObservation *LiveObservation | |
| 122 | + | OpenOperations []Operation | |
| 123 | + | Terminal string | |
| 124 | + | Result *ResultFact | |
| 125 | + | LastRequest *RequestFact | |
| 126 | + | } | |
| 127 | + | ||
| 128 | + | // Summary returns the most specific factual activity or terminal result fact. | |
| 129 | + | func (v AgentTraceView) Summary() string { | |
| 130 | + | if v.Terminal != "" { | |
| 131 | + | if v.Result != nil && v.Result.Available { | |
| 132 | + | if v.Result.Delivered { | |
| 133 | + | return v.Terminal + " · result delivered" | |
| 134 | + | } | |
| 135 | + | if v.Result.Failed { | |
| 136 | + | return v.Terminal + " · result delivery failed" | |
| 137 | + | } | |
| 138 | + | return v.Terminal + " · result available" | |
| 139 | + | } | |
| 140 | + | return v.Terminal | |
| 141 | + | } | |
| 142 | + | if primary, ok := v.Primary(); ok { | |
| 143 | + | if observation := v.ObservationFor(primary); observation != nil { | |
| 144 | + | return primary.Label() + " · " + observation.Label() | |
| 145 | + | } | |
| 146 | + | return primary.Label() | |
| 147 | + | } | |
| 148 | + | if v.LastEvent.Type != "" { | |
| 149 | + | return "last event " + v.LastEvent.Type | |
| 150 | + | } | |
| 151 | + | return "no observable operation" | |
| 43 | 152 | } | |
| 44 | 153 | ||
| 45 | 154 | // ProjectTrace reduces a chronologically ordered trace into per-agent facts. | |
| 46 | - | // Unknown events are retained as LastEvent but never guessed into an operation. | |
| 155 | + | // Unknown worker protocol events remain factual observations and are never | |
| 156 | + | // guessed into operations. | |
| 47 | 157 | func ProjectTrace(events []TraceEvent) map[string]AgentTraceView { | |
| 48 | - | views := make(map[string]AgentTraceView) | |
| 49 | - | open := make(map[string]map[string]Operation) | |
| 158 | + | projection := newTraceProjection() | |
| 50 | 159 | for _, event := range events { | |
| 51 | - | if event.AgentID == "" { | |
| 52 | - | continue | |
| 160 | + | projection.apply(event) | |
| 161 | + | } | |
| 162 | + | return projection.snapshot() | |
| 163 | + | } | |
| 164 | + | ||
| 165 | + | // traceProjection incrementally derives user-facing facts from the trace. It | |
| 166 | + | // deliberately stores only compact facts and currently open operations, so a | |
| 167 | + | // TraceWriter can evict old raw events without losing the start boundary of a | |
| 168 | + | // long-running delegated turn. | |
| 169 | + | type traceProjection struct { | |
| 170 | + | views map[string]AgentTraceView | |
| 171 | + | open map[string]map[string]Operation | |
| 172 | + | } | |
| 173 | + | ||
| 174 | + | func newTraceProjection() *traceProjection { | |
| 175 | + | return &traceProjection{ | |
| 176 | + | views: make(map[string]AgentTraceView), | |
| 177 | + | open: make(map[string]map[string]Operation), | |
| 178 | + | } | |
| 179 | + | } | |
| 180 | + | ||
| 181 | + | func (p *traceProjection) apply(event TraceEvent) { | |
| 182 | + | if event.AgentID == "" { | |
| 183 | + | return | |
| 184 | + | } | |
| 185 | + | view := p.views[event.AgentID] | |
| 186 | + | view.AgentID = event.AgentID | |
| 187 | + | view.LastEvent = event | |
| 188 | + | key, starts, ends, terminal := traceBoundary(event) | |
| 189 | + | var endedOperation Operation | |
| 190 | + | var endedOperationFound bool | |
| 191 | + | if ends { | |
| 192 | + | endedOperation, endedOperationFound = p.open[event.AgentID][key] | |
| 193 | + | } | |
| 194 | + | if starts { | |
| 195 | + | if event.Type == "provider.request.started" { | |
| 196 | + | view.LastRequest = requestFact(event, "open") | |
| 53 | 197 | } | |
| 54 | - | view := views[event.AgentID] | |
| 55 | - | view.AgentID = event.AgentID | |
| 56 | - | view.LastEvent = event | |
| 57 | - | key, starts, ends, terminal := traceBoundary(event) | |
| 58 | - | if starts { | |
| 59 | - | if open[event.AgentID] == nil { | |
| 60 | - | open[event.AgentID] = make(map[string]Operation) | |
| 198 | + | if p.open[event.AgentID] == nil { | |
| 199 | + | p.open[event.AgentID] = make(map[string]Operation) | |
| 200 | + | } | |
| 201 | + | callID, _ := event.Data["call_id"].(string) | |
| 202 | + | name, _ := event.Data["name"].(string) | |
| 203 | + | provider, _ := event.Data["provider"].(string) | |
| 204 | + | model, _ := event.Data["model"].(string) | |
| 205 | + | attempt := traceInt(event.Data["attempt"]) | |
| 206 | + | maxAttempts := traceInt(event.Data["max_attempts"]) | |
| 207 | + | step := traceInt(event.Data["step"]) | |
| 208 | + | p.open[event.AgentID][key] = Operation{Type: event.Type, AgentID: event.AgentID, TurnID: event.TurnID, CallID: callID, Name: name, Provider: provider, Model: model, Step: step, Attempt: attempt, MaxAttempts: maxAttempts, StartedAt: event.Timestamp} | |
| 209 | + | } | |
| 210 | + | if ends { | |
| 211 | + | if event.Type == "provider.request.finished" || event.Type == "provider.request.failed" || event.Type == "provider.request.cancelled" || event.Type == "provider.request.retry.scheduled" { | |
| 212 | + | if active, found := p.open[event.AgentID][key]; found { | |
| 213 | + | outcome := strings.TrimPrefix(event.Type, "provider.request.") | |
| 214 | + | request := requestFact(event, outcome) | |
| 215 | + | request.Provider, request.Model = active.Provider, active.Model | |
| 216 | + | request.Attempt, request.MaxAttempts = active.Attempt, active.MaxAttempts | |
| 217 | + | request.StartedAt = active.StartedAt | |
| 218 | + | view.LastRequest = request | |
| 61 | 219 | } | |
| 62 | - | callID, _ := event.Data["call_id"].(string) | |
| 63 | - | open[event.AgentID][key] = Operation{Type: event.Type, AgentID: event.AgentID, TurnID: event.TurnID, CallID: callID, StartedAt: event.Timestamp} | |
| 64 | 220 | } | |
| 65 | - | if ends { | |
| 66 | - | delete(open[event.AgentID], key) | |
| 221 | + | delete(p.open[event.AgentID], key) | |
| 222 | + | } | |
| 223 | + | if observation := traceObservation(event); observation != nil { | |
| 224 | + | if observation.Type == "tool.finished" && observation.Name == "" { | |
| 225 | + | // tool_result carries only the call ID. Recover the safe tool name | |
| 226 | + | // from its matching open execution boundary; an unscoped generic | |
| 227 | + | // "tool finished" fact is too low-value for the live indicator. | |
| 228 | + | if endedOperationFound && endedOperation.Type == "tool.started" && endedOperation.Name != "" { | |
| 229 | + | observation.Name = endedOperation.Name | |
| 230 | + | } else { | |
| 231 | + | observation = nil | |
| 232 | + | } | |
| 67 | 233 | } | |
| 68 | - | if terminal != "" { | |
| 69 | - | view.Terminal = terminal | |
| 70 | - | delete(open, event.AgentID) | |
| 234 | + | if observation != nil { | |
| 235 | + | view.LastObservation = observation | |
| 71 | 236 | } | |
| 72 | - | views[event.AgentID] = view | |
| 73 | 237 | } | |
| 74 | - | for agentID, operations := range open { | |
| 75 | - | view := views[agentID] | |
| 238 | + | if result := traceResultFact(event, view.Result); result != nil { | |
| 239 | + | view.Result = result | |
| 240 | + | } | |
| 241 | + | if terminal != "" { | |
| 242 | + | view.Terminal = terminal | |
| 243 | + | delete(p.open, event.AgentID) | |
| 244 | + | } | |
| 245 | + | p.views[event.AgentID] = view | |
| 246 | + | } | |
| 247 | + | ||
| 248 | + | func (p *traceProjection) snapshot() map[string]AgentTraceView { | |
| 249 | + | views := make(map[string]AgentTraceView, len(p.views)) | |
| 250 | + | for agentID, stored := range p.views { | |
| 251 | + | view := cloneAgentTraceView(stored) | |
| 252 | + | operations := p.open[agentID] | |
| 76 | 253 | for _, operation := range operations { | |
| 77 | 254 | view.OpenOperations = append(view.OpenOperations, operation) | |
| 78 | 255 | } | |
| 79 | 256 | sort.Slice(view.OpenOperations, func(i, j int) bool { | |
| 80 | 257 | left, right := view.OpenOperations[i], view.OpenOperations[j] | |
| 258 | + | if operationPriority(left) != operationPriority(right) { | |
| 259 | + | return operationPriority(left) < operationPriority(right) | |
| 260 | + | } | |
| 81 | 261 | if !left.StartedAt.Equal(right.StartedAt) { | |
| 82 | 262 | return left.StartedAt.Before(right.StartedAt) | |
| 83 | 263 | } | |
| @@ -94,6 +274,160 @@ func ProjectTrace(events []TraceEvent) map[string]AgentTraceView { | |||
| 94 | 274 | return views | |
| 95 | 275 | } | |
| 96 | 276 | ||
| 277 | + | func cloneAgentTraceView(view AgentTraceView) AgentTraceView { | |
| 278 | + | view.OpenOperations = nil | |
| 279 | + | if view.LastObservation != nil { | |
| 280 | + | observation := *view.LastObservation | |
| 281 | + | view.LastObservation = &observation | |
| 282 | + | } | |
| 283 | + | if view.Result != nil { | |
| 284 | + | result := *view.Result | |
| 285 | + | view.Result = &result | |
| 286 | + | } | |
| 287 | + | if view.LastRequest != nil { | |
| 288 | + | request := *view.LastRequest | |
| 289 | + | view.LastRequest = &request | |
| 290 | + | } | |
| 291 | + | return view | |
| 292 | + | } | |
| 293 | + | ||
| 294 | + | // Primary returns the highest-priority factual operation from the ordered | |
| 295 | + | // open-operation projection. | |
| 296 | + | func (v AgentTraceView) Primary() (Operation, bool) { | |
| 297 | + | if len(v.OpenOperations) == 0 { | |
| 298 | + | return Operation{}, false | |
| 299 | + | } | |
| 300 | + | return v.OpenOperations[0], true | |
| 301 | + | } | |
| 302 | + | ||
| 303 | + | // TurnStartedAt returns the factual start of the open delegated turn owning | |
| 304 | + | // operation. It is stable across nested provider requests and tool calls. | |
| 305 | + | func (v AgentTraceView) TurnStartedAt(operation Operation) time.Time { | |
| 306 | + | if operation.TurnID == "" { | |
| 307 | + | return time.Time{} | |
| 308 | + | } | |
| 309 | + | for _, candidate := range v.OpenOperations { | |
| 310 | + | if candidate.Type == "turn.started" && candidate.TurnID == operation.TurnID { | |
| 311 | + | return candidate.StartedAt | |
| 312 | + | } | |
| 313 | + | } | |
| 314 | + | return time.Time{} | |
| 315 | + | } | |
| 316 | + | ||
| 317 | + | // ObservationFor returns the last user-facing observation only when it belongs | |
| 318 | + | // to this operation's turn and scope. Tool observations are shown for their | |
| 319 | + | // matching open tool and may remain as a factual recent fact under the next | |
| 320 | + | // provider request in that turn. Provider-stream observations never decorate | |
| 321 | + | // a tool operation, and another tool's observation never crosses call scope. | |
| 322 | + | func (v AgentTraceView) ObservationFor(operation Operation) *LiveObservation { | |
| 323 | + | observation := v.LastObservation | |
| 324 | + | if observation == nil || operation.TurnID == "" || observation.TurnID == "" || operation.TurnID != observation.TurnID { | |
| 325 | + | return nil | |
| 326 | + | } | |
| 327 | + | switch operation.Type { | |
| 328 | + | case "tool.started": | |
| 329 | + | if !isToolObservation(observation.Type) || operation.CallID == "" || observation.CallID == "" || operation.CallID != observation.CallID { | |
| 330 | + | return nil | |
| 331 | + | } | |
| 332 | + | case "provider.request.started": | |
| 333 | + | // The provider request is the current primary operation. A completed | |
| 334 | + | // tool from this turn remains useful context without pretending the | |
| 335 | + | // closed tool is still running. | |
| 336 | + | case "turn.started": | |
| 337 | + | // Keep even a very short tool perceptible after its factual end boundary | |
| 338 | + | // and before the next provider request boundary arrives. Other call-scoped | |
| 339 | + | // facts do not decorate the broader turn operation. | |
| 340 | + | if observation.CallID != "" && observation.Type != "tool.finished" { | |
| 341 | + | return nil | |
| 342 | + | } | |
| 343 | + | default: | |
| 344 | + | if observation.CallID != "" { | |
| 345 | + | return nil | |
| 346 | + | } | |
| 347 | + | } | |
| 348 | + | return observation | |
| 349 | + | } | |
| 350 | + | ||
| 351 | + | func isToolObservation(typeName string) bool { | |
| 352 | + | switch typeName { | |
| 353 | + | case "tool.output.observed", "tool.finished": | |
| 354 | + | return true | |
| 355 | + | default: | |
| 356 | + | return false | |
| 357 | + | } | |
| 358 | + | } | |
| 359 | + | ||
| 360 | + | func operationPriority(o Operation) int { | |
| 361 | + | switch o.Type { | |
| 362 | + | case "tool.started": | |
| 363 | + | return 0 | |
| 364 | + | case "provider.request.started": | |
| 365 | + | return 1 | |
| 366 | + | case "agent.wait.started": | |
| 367 | + | return 2 | |
| 368 | + | default: | |
| 369 | + | return 3 | |
| 370 | + | } | |
| 371 | + | } | |
| 372 | + | ||
| 373 | + | func traceObservation(event TraceEvent) *LiveObservation { | |
| 374 | + | typeName := event.Type | |
| 375 | + | source, _ := event.Data["source_event"].(string) | |
| 376 | + | if source == "" { | |
| 377 | + | source, _ = event.Data["worker_event"].(string) | |
| 378 | + | } | |
| 379 | + | name, _ := event.Data["name"].(string) | |
| 380 | + | callID, _ := event.Data["call_id"].(string) | |
| 381 | + | switch typeName { | |
| 382 | + | case "assistant.stream.observed", "reasoning.stream.observed", "tool.output.observed", "tool.finished": | |
| 383 | + | return &LiveObservation{Type: typeName, Source: source, At: event.Timestamp, TurnID: event.TurnID, CallID: callID, Name: name} | |
| 384 | + | } | |
| 385 | + | return nil | |
| 386 | + | } | |
| 387 | + | ||
| 388 | + | func traceInt(value any) int { | |
| 389 | + | switch value := value.(type) { | |
| 390 | + | case int: | |
| 391 | + | return value | |
| 392 | + | case float64: | |
| 393 | + | return int(value) | |
| 394 | + | default: | |
| 395 | + | return 0 | |
| 396 | + | } | |
| 397 | + | } | |
| 398 | + | ||
| 399 | + | func requestFact(event TraceEvent, outcome string) *RequestFact { | |
| 400 | + | providerName, _ := event.Data["provider"].(string) | |
| 401 | + | model, _ := event.Data["model"].(string) | |
| 402 | + | attempt := traceInt(event.Data["attempt"]) | |
| 403 | + | maxAttempts := traceInt(event.Data["max_attempts"]) | |
| 404 | + | errorCode, _ := event.Data["error_code"].(string) | |
| 405 | + | return &RequestFact{TurnID: event.TurnID, Provider: providerName, Model: model, Attempt: attempt, MaxAttempts: maxAttempts, Outcome: outcome, ErrorCode: errorCode, StartedAt: event.Timestamp, EndedAt: event.Timestamp} | |
| 406 | + | } | |
| 407 | + | ||
| 408 | + | func traceResultFact(event TraceEvent, previous *ResultFact) *ResultFact { | |
| 409 | + | if event.Type != "result.available" && event.Type != "result.delivered" && event.Type != "result.delivery.failed" { | |
| 410 | + | return previous | |
| 411 | + | } | |
| 412 | + | result := ResultFact{} | |
| 413 | + | if previous != nil { | |
| 414 | + | result = *previous | |
| 415 | + | } | |
| 416 | + | if ref, _ := event.Data["ref"].(string); ref != "" { | |
| 417 | + | result.Ref = ref | |
| 418 | + | } | |
| 419 | + | result.At = event.Timestamp | |
| 420 | + | switch event.Type { | |
| 421 | + | case "result.available": | |
| 422 | + | result.Available, result.Delivered, result.Failed = true, false, false | |
| 423 | + | case "result.delivered": | |
| 424 | + | result.Available, result.Delivered, result.Failed = true, true, false | |
| 425 | + | case "result.delivery.failed": | |
| 426 | + | result.Available, result.Delivered, result.Failed = true, false, true | |
| 427 | + | } | |
| 428 | + | return &result | |
| 429 | + | } | |
| 430 | + | ||
| 97 | 431 | func traceBoundary(event TraceEvent) (key string, starts, ends bool, terminal string) { | |
| 98 | 432 | callID, _ := event.Data["call_id"].(string) | |
| 99 | 433 | key = event.Type + ":" + event.TurnID + ":" + callID | |
| @@ -106,6 +440,14 @@ func traceBoundary(event TraceEvent) (key string, starts, ends bool, terminal st | |||
| 106 | 440 | return "tool:" + event.TurnID + ":" + callID, true, false, "" | |
| 107 | 441 | case "tool.finished", "tool.failed", "tool.cancelled": | |
| 108 | 442 | return "tool:" + event.TurnID + ":" + callID, false, true, "" | |
| 443 | + | case "provider.request.started": | |
| 444 | + | return "provider:" + event.TurnID, true, false, "" | |
| 445 | + | case "provider.request.finished", "provider.request.failed", "provider.request.cancelled", "provider.request.retry.scheduled": | |
| 446 | + | return "provider:" + event.TurnID, false, true, "" | |
| 447 | + | case "agent.wait.started": | |
| 448 | + | return "wait:" + event.TurnID, true, false, "" | |
| 449 | + | case "agent.wait.finished": | |
| 450 | + | return "wait:" + event.TurnID, false, true, "" | |
| 109 | 451 | case "agent.finished": | |
| 110 | 452 | return key, false, false, "completed" | |
| 111 | 453 | case "agent.failed": | |
M packages/agent/subagents/projection_test.go +217 -0
| @@ -1,10 +1,115 @@ | |||
| 1 | 1 | package subagents | |
| 2 | 2 | ||
| 3 | 3 | import ( | |
| 4 | + | "os" | |
| 5 | + | "path/filepath" | |
| 6 | + | "reflect" | |
| 4 | 7 | "testing" | |
| 5 | 8 | "time" | |
| 6 | 9 | ) | |
| 7 | 10 | ||
| 11 | + | func primaryOperation(t testing.TB, view AgentTraceView) *Operation { | |
| 12 | + | t.Helper() | |
| 13 | + | operation, ok := view.Primary() | |
| 14 | + | if !ok { | |
| 15 | + | t.Fatal("trace view has no primary operation") | |
| 16 | + | } | |
| 17 | + | return &operation | |
| 18 | + | } | |
| 19 | + | ||
| 20 | + | func TestProjectTraceRetainsJSONDecodedRequestAttempts(t *testing.T) { | |
| 21 | + | bundle := t.TempDir() | |
| 22 | + | if err := os.WriteFile(filepath.Join(bundle, "manifest.json"), []byte(`{"version":1,"trace_file":"trace.jsonl"}`), 0o600); err != nil { | |
| 23 | + | t.Fatal(err) | |
| 24 | + | } | |
| 25 | + | line := `{"seq":1,"timestamp":"2026-08-12T07:00:00Z","type":"provider.request.started","agent_id":"agent-1","turn_id":"turn-1","data":{"attempt":2,"max_attempts":6}}` + "\n" | |
| 26 | + | if err := os.WriteFile(filepath.Join(bundle, "trace.jsonl"), []byte(line), 0o600); err != nil { | |
| 27 | + | t.Fatal(err) | |
| 28 | + | } | |
| 29 | + | events, err := ReadTrace(bundle) | |
| 30 | + | if err != nil { | |
| 31 | + | t.Fatal(err) | |
| 32 | + | } | |
| 33 | + | operation := primaryOperation(t, ProjectTrace(events)["agent-1"]) | |
| 34 | + | if operation == nil || operation.Attempt != 2 || operation.MaxAttempts != 6 { | |
| 35 | + | t.Fatalf("projected operation = %#v, want attempt 2/6", operation) | |
| 36 | + | } | |
| 37 | + | } | |
| 38 | + | ||
| 39 | + | func TestProjectTraceIgnoresProtocolFactsAndShowsCompletedToolDuringNextProviderRequest(t *testing.T) { | |
| 40 | + | started := time.Date(2026, 8, 12, 7, 0, 0, 0, time.UTC) | |
| 41 | + | view := ProjectTrace([]TraceEvent{ | |
| 42 | + | {Timestamp: started, Type: "turn.started", AgentID: "agent-1", TurnID: "turn-1"}, | |
| 43 | + | {Timestamp: started.Add(time.Second), Type: "tool.started", AgentID: "agent-1", TurnID: "turn-1", Data: map[string]any{"call_id": "call-1", "name": "read"}}, | |
| 44 | + | {Timestamp: started.Add(2 * time.Second), Type: "tool.finished", AgentID: "agent-1", TurnID: "turn-1", Data: map[string]any{"call_id": "call-1", "name": "read"}}, | |
| 45 | + | {Timestamp: started.Add(2500 * time.Millisecond), Type: "worker.protocol.observed", AgentID: "agent-1", TurnID: "turn-1", Data: map[string]any{"source_event": "tool.use.args"}}, | |
| 46 | + | {Timestamp: started.Add(3 * time.Second), Type: "provider.request.started", AgentID: "agent-1", TurnID: "turn-1"}, | |
| 47 | + | })["agent-1"] | |
| 48 | + | if primaryOperation(t, view) == nil || primaryOperation(t, view).Type != "provider.request.started" { | |
| 49 | + | t.Fatalf("primary operation = %#v", primaryOperation(t, view)) | |
| 50 | + | } | |
| 51 | + | observation := view.ObservationFor(*primaryOperation(t, view)) | |
| 52 | + | if observation == nil || observation.Label() != "finished read" { | |
| 53 | + | t.Fatalf("observation = %#v, want finished read", observation) | |
| 54 | + | } | |
| 55 | + | if view.LastObservation != observation { | |
| 56 | + | t.Fatalf("protocol fact replaced user-facing observation: %#v", view.LastObservation) | |
| 57 | + | } | |
| 58 | + | } | |
| 59 | + | ||
| 60 | + | func TestProjectTraceSuppressesGenericUnmatchedToolFinished(t *testing.T) { | |
| 61 | + | view := ProjectTrace([]TraceEvent{ | |
| 62 | + | {Type: "turn.started", AgentID: "agent-1", TurnID: "turn-1"}, | |
| 63 | + | {Type: "tool.finished", AgentID: "agent-1", TurnID: "turn-1", Data: map[string]any{"call_id": "unknown"}}, | |
| 64 | + | })["agent-1"] | |
| 65 | + | if view.LastObservation != nil { | |
| 66 | + | t.Fatalf("generic tool-finished observation = %#v, want omitted", view.LastObservation) | |
| 67 | + | } | |
| 68 | + | } | |
| 69 | + | ||
| 70 | + | func TestProjectTraceKeepsShortCompletedToolVisibleBeforeNextRequest(t *testing.T) { | |
| 71 | + | started := time.Date(2026, 8, 12, 7, 0, 0, 0, time.UTC) | |
| 72 | + | view := ProjectTrace([]TraceEvent{ | |
| 73 | + | {Timestamp: started, Type: "turn.started", AgentID: "agent-1", TurnID: "turn-1"}, | |
| 74 | + | {Timestamp: started.Add(time.Second), Type: "tool.started", AgentID: "agent-1", TurnID: "turn-1", Data: map[string]any{"call_id": "call-1", "name": "read"}}, | |
| 75 | + | {Timestamp: started.Add(time.Second + time.Millisecond), Type: "tool.finished", AgentID: "agent-1", TurnID: "turn-1", Data: map[string]any{"call_id": "call-1", "name": "read"}}, | |
| 76 | + | })["agent-1"] | |
| 77 | + | if primaryOperation(t, view) == nil || primaryOperation(t, view).Type != "turn.started" { | |
| 78 | + | t.Fatalf("primary operation = %#v, want turn", primaryOperation(t, view)) | |
| 79 | + | } | |
| 80 | + | if observation := view.ObservationFor(*primaryOperation(t, view)); observation == nil || observation.Label() != "finished read" { | |
| 81 | + | t.Fatalf("completed tool observation = %#v, want finished read", observation) | |
| 82 | + | } | |
| 83 | + | } | |
| 84 | + | ||
| 85 | + | func TestProjectTraceDoesNotDecorateToolWithProviderStream(t *testing.T) { | |
| 86 | + | started := time.Date(2026, 8, 12, 7, 0, 0, 0, time.UTC) | |
| 87 | + | view := ProjectTrace([]TraceEvent{ | |
| 88 | + | {Timestamp: started, Type: "turn.started", AgentID: "agent-1", TurnID: "turn-1"}, | |
| 89 | + | {Timestamp: started.Add(time.Second), Type: "tool.started", AgentID: "agent-1", TurnID: "turn-1", Data: map[string]any{"call_id": "call-1", "name": "read"}}, | |
| 90 | + | {Timestamp: started.Add(2 * time.Second), Type: "assistant.stream.observed", AgentID: "agent-1", TurnID: "turn-1"}, | |
| 91 | + | })["agent-1"] | |
| 92 | + | if primaryOperation(t, view) == nil || primaryOperation(t, view).Type != "tool.started" { | |
| 93 | + | t.Fatalf("primary operation = %#v, want tool", primaryOperation(t, view)) | |
| 94 | + | } | |
| 95 | + | if observation := view.ObservationFor(*primaryOperation(t, view)); observation != nil { | |
| 96 | + | t.Fatalf("provider stream decorated tool operation: %#v", observation) | |
| 97 | + | } | |
| 98 | + | } | |
| 99 | + | ||
| 100 | + | func TestAgentTraceViewTurnStartedAtMatchesOwningTurn(t *testing.T) { | |
| 101 | + | first := time.Date(2026, 8, 12, 7, 0, 0, 0, time.UTC) | |
| 102 | + | second := first.Add(time.Minute) | |
| 103 | + | view := AgentTraceView{OpenOperations: []Operation{ | |
| 104 | + | {Type: "turn.started", TurnID: "turn-1", StartedAt: first}, | |
| 105 | + | {Type: "turn.started", TurnID: "turn-2", StartedAt: second}, | |
| 106 | + | }} | |
| 107 | + | operation := Operation{Type: "provider.request.started", TurnID: "turn-2", StartedAt: second.Add(time.Second)} | |
| 108 | + | if got := view.TurnStartedAt(operation); !got.Equal(second) { | |
| 109 | + | t.Fatalf("turn start = %v, want %v", got, second) | |
| 110 | + | } | |
| 111 | + | } | |
| 112 | + | ||
| 8 | 113 | func TestProjectTraceReportsOpenOperationInsteadOfLifecycleState(t *testing.T) { | |
| 9 | 114 | started := time.Date(2026, 8, 12, 7, 0, 0, 0, time.UTC) | |
| 10 | 115 | views := ProjectTrace([]TraceEvent{ | |
| @@ -56,6 +161,118 @@ func TestProjectTraceClosesFailedToolByCallID(t *testing.T) { | |||
| 56 | 161 | } | |
| 57 | 162 | } | |
| 58 | 163 | ||
| 164 | + | func TestProjectTraceSelectsSpecificActivityAndProjectsResultDelivery(t *testing.T) { | |
| 165 | + | started := time.Date(2026, 8, 12, 7, 0, 0, 0, time.UTC) | |
| 166 | + | view := ProjectTrace([]TraceEvent{ | |
| 167 | + | {Timestamp: started, Type: "turn.started", AgentID: "agent-1", TurnID: "turn-1"}, | |
| 168 | + | {Timestamp: started.Add(time.Second), Type: "tool.started", AgentID: "agent-1", TurnID: "turn-1", Data: map[string]any{"call_id": "one", "name": "bash"}}, | |
| 169 | + | {Timestamp: started.Add(2 * time.Second), Type: "assistant.stream.observed", AgentID: "agent-1", TurnID: "turn-1", Data: map[string]any{"source_event": "message.delta"}}, | |
| 170 | + | {Timestamp: started.Add(3 * time.Second), Type: "result.available", AgentID: "agent-1", TurnID: "turn-1", Data: map[string]any{"ref": "subagent://agent-1/result"}}, | |
| 171 | + | {Timestamp: started.Add(4 * time.Second), Type: "result.delivered", AgentID: "agent-1", TurnID: "turn-1", Data: map[string]any{"ref": "subagent://agent-1/result"}}, | |
| 172 | + | })["agent-1"] | |
| 173 | + | if primaryOperation(t, view) == nil || primaryOperation(t, view).Type != "tool.started" || primaryOperation(t, view).Name != "bash" { | |
| 174 | + | t.Fatalf("primary operation = %#v, want bash tool", primaryOperation(t, view)) | |
| 175 | + | } | |
| 176 | + | if view.LastObservation == nil || view.LastObservation.Type != "assistant.stream.observed" || view.LastObservation.Label() != "assistant streaming" { | |
| 177 | + | t.Fatalf("last observation = %#v", view.LastObservation) | |
| 178 | + | } | |
| 179 | + | if view.Result == nil || !view.Result.Available || !view.Result.Delivered || view.Result.Ref != "subagent://agent-1/result" { | |
| 180 | + | t.Fatalf("result = %#v", view.Result) | |
| 181 | + | } | |
| 182 | + | if got, want := view.Summary(), "running bash"; got != want { | |
| 183 | + | t.Fatalf("summary = %q, want %q", got, want) | |
| 184 | + | } | |
| 185 | + | } | |
| 186 | + | ||
| 187 | + | func TestOperationLabelUsesHumanFacingActivity(t *testing.T) { | |
| 188 | + | cases := []struct { | |
| 189 | + | operation Operation | |
| 190 | + | want string | |
| 191 | + | }{ | |
| 192 | + | {operation: Operation{Type: "provider.request.started"}, want: "waiting for model response"}, | |
| 193 | + | {operation: Operation{Type: "tool.started", Name: "bash"}, want: "running bash"}, | |
| 194 | + | {operation: Operation{Type: "turn.started"}, want: "processing turn"}, | |
| 195 | + | } | |
| 196 | + | for _, tc := range cases { | |
| 197 | + | if got := tc.operation.Label(); got != tc.want { | |
| 198 | + | t.Fatalf("Label(%q) = %q, want %q", tc.operation.Type, got, tc.want) | |
| 199 | + | } | |
| 200 | + | } | |
| 201 | + | } | |
| 202 | + | ||
| 203 | + | func TestProjectTraceRetainsSafeProviderAttemptDiagnostic(t *testing.T) { | |
| 204 | + | started := time.Date(2026, 8, 12, 7, 0, 0, 0, time.UTC) | |
| 205 | + | view := ProjectTrace([]TraceEvent{ | |
| 206 | + | {Timestamp: started, Type: "provider.request.started", AgentID: "agent-1", TurnID: "turn-1", Data: map[string]any{"provider": "openai-codex", "model": "gpt-5.6", "attempt": 2, "max_attempts": 4}}, | |
| 207 | + | {Timestamp: started.Add(time.Second), Type: "provider.request.failed", AgentID: "agent-1", TurnID: "turn-1", Data: map[string]any{"error_code": "deadline_exceeded"}}, | |
| 208 | + | })["agent-1"] | |
| 209 | + | if len(view.OpenOperations) != 0 { | |
| 210 | + | t.Fatalf("open operations = %#v, want none", view.OpenOperations) | |
| 211 | + | } | |
| 212 | + | if got, want := view.LastRequest, (&RequestFact{TurnID: "turn-1", Provider: "openai-codex", Model: "gpt-5.6", Attempt: 2, MaxAttempts: 4, Outcome: "failed", ErrorCode: "deadline_exceeded", StartedAt: started, EndedAt: started.Add(time.Second)}); !reflect.DeepEqual(got, want) { | |
| 213 | + | t.Fatalf("request diagnostic = %#v, want %#v", got, want) | |
| 214 | + | } | |
| 215 | + | } | |
| 216 | + | ||
| 217 | + | func TestProjectTraceOmitsObservationForAnotherToolInSameTurn(t *testing.T) { | |
| 218 | + | started := time.Date(2026, 8, 12, 7, 0, 0, 0, time.UTC) | |
| 219 | + | view := ProjectTrace([]TraceEvent{ | |
| 220 | + | {Timestamp: started, Type: "turn.started", AgentID: "agent-1", TurnID: "turn-1"}, | |
| 221 | + | {Timestamp: started.Add(time.Second), Type: "tool.started", AgentID: "agent-1", TurnID: "turn-1", Data: map[string]any{"call_id": "one", "name": "bash"}}, | |
| 222 | + | {Timestamp: started.Add(2 * time.Second), Type: "tool.started", AgentID: "agent-1", TurnID: "turn-1", Data: map[string]any{"call_id": "two", "name": "read"}}, | |
| 223 | + | {Timestamp: started.Add(3 * time.Second), Type: "tool.output.observed", AgentID: "agent-1", TurnID: "turn-1", Data: map[string]any{"call_id": "two", "name": "read"}}, | |
| 224 | + | })["agent-1"] | |
| 225 | + | if primaryOperation(t, view) == nil || primaryOperation(t, view).CallID != "one" { | |
| 226 | + | t.Fatalf("primary operation = %#v, want call one", primaryOperation(t, view)) | |
| 227 | + | } | |
| 228 | + | if observation := view.ObservationFor(*primaryOperation(t, view)); observation != nil { | |
| 229 | + | t.Fatalf("another tool's observation = %#v, want omitted", observation) | |
| 230 | + | } | |
| 231 | + | if got, want := view.Summary(), "running bash"; got != want { | |
| 232 | + | t.Fatalf("summary = %q, want %q", got, want) | |
| 233 | + | } | |
| 234 | + | } | |
| 235 | + | ||
| 236 | + | func TestProjectTraceOmitsPriorTurnObservation(t *testing.T) { | |
| 237 | + | started := time.Date(2026, 8, 12, 7, 0, 0, 0, time.UTC) | |
| 238 | + | view := ProjectTrace([]TraceEvent{ | |
| 239 | + | {Timestamp: started, Type: "assistant.stream.observed", AgentID: "agent-1", TurnID: "turn-1"}, | |
| 240 | + | {Timestamp: started.Add(time.Second), Type: "turn.started", AgentID: "agent-1", TurnID: "turn-2"}, | |
| 241 | + | })["agent-1"] | |
| 242 | + | if view.LastObservation == nil { | |
| 243 | + | t.Fatal("last observation missing") | |
| 244 | + | } | |
| 245 | + | if primaryOperation(t, view) == nil { | |
| 246 | + | t.Fatal("primary operation missing") | |
| 247 | + | } | |
| 248 | + | if observation := view.ObservationFor(*primaryOperation(t, view)); observation != nil { | |
| 249 | + | t.Fatalf("prior turn observation = %#v, want omitted", observation) | |
| 250 | + | } | |
| 251 | + | if got, want := view.Summary(), "processing turn"; got != want { | |
| 252 | + | t.Fatalf("summary = %q, want %q", got, want) | |
| 253 | + | } | |
| 254 | + | } | |
| 255 | + | ||
| 256 | + | func TestProjectTraceNewAvailableResultResetsDelivery(t *testing.T) { | |
| 257 | + | view := ProjectTrace([]TraceEvent{ | |
| 258 | + | {Type: "result.delivered", AgentID: "agent-1", Data: map[string]any{"ref": "subagent://agent-1/result-1"}}, | |
| 259 | + | {Type: "result.available", AgentID: "agent-1", Data: map[string]any{"ref": "subagent://agent-1/result-2"}}, | |
| 260 | + | })["agent-1"] | |
| 261 | + | if view.Result == nil || !view.Result.Available || view.Result.Delivered || view.Result.Failed || view.Result.Ref != "subagent://agent-1/result-2" { | |
| 262 | + | t.Fatalf("result = %#v", view.Result) | |
| 263 | + | } | |
| 264 | + | } | |
| 265 | + | ||
| 266 | + | func TestProjectTraceDeliveryFailureClearsDelivered(t *testing.T) { | |
| 267 | + | view := ProjectTrace([]TraceEvent{ | |
| 268 | + | {Type: "result.delivered", AgentID: "agent-1", Data: map[string]any{"ref": "subagent://agent-1/result"}}, | |
| 269 | + | {Type: "result.delivery.failed", AgentID: "agent-1", Data: map[string]any{"ref": "subagent://agent-1/result"}}, | |
| 270 | + | })["agent-1"] | |
| 271 | + | if view.Result == nil || !view.Result.Available || view.Result.Delivered || !view.Result.Failed { | |
| 272 | + | t.Fatalf("result = %#v, want available undelivered failure", view.Result) | |
| 273 | + | } | |
| 274 | + | } | |
| 275 | + | ||
| 59 | 276 | func TestProjectTraceFailedAgentClosesOpenOperations(t *testing.T) { | |
| 60 | 277 | view := ProjectTrace([]TraceEvent{ | |
| 61 | 278 | {Type: "tool.started", AgentID: "agent-1", TurnID: "turn-1", Data: map[string]any{"call_id": "one"}}, | |
M packages/agent/subagents/protocol.go +1 -1
| @@ -600,7 +600,7 @@ type AgentHeartbeatPayload struct { | |||
| 600 | 600 | } | |
| 601 | 601 | type TurnStartPayload struct { | |
| 602 | 602 | Prompt string `json:"prompt"` | |
| 603 | - | // NewRun starts a fresh max_turns budget while retaining the worker's | |
| 603 | + | // NewRun starts a fresh observable run while retaining the worker's | |
| 604 | 604 | // existing session and transcript. It is set only for explicit | |
| 605 | 605 | // subagent_resume follow-ups. | |
| 606 | 606 | NewRun bool `json:"new_run,omitempty"` | |
M packages/agent/subagents/requirement.go +45 -11
| @@ -2,6 +2,7 @@ package subagents | |||
| 2 | 2 | ||
| 3 | 3 | import ( | |
| 4 | 4 | "context" | |
| 5 | + | "errors" | |
| 5 | 6 | "fmt" | |
| 6 | 7 | "sort" | |
| 7 | 8 | "strconv" | |
| @@ -26,12 +27,13 @@ const ( | |||
| 26 | 27 | // delegated message turn rather than a process attempt because resumable | |
| 27 | 28 | // workers may execute several turns in one process. | |
| 28 | 29 | type RequirementSnapshot struct { | |
| 29 | - | Required bool `json:"required"` | |
| 30 | - | State RequirementState `json:"state,omitempty"` | |
| 31 | - | TargetTurn int `json:"target_turn,omitempty"` | |
| 32 | - | UpdatedAt time.Time `json:"updated_at,omitempty"` | |
| 33 | - | ErrorCode string `json:"error_code,omitempty"` | |
| 34 | - | Notified bool `json:"notified,omitempty"` | |
| 30 | + | Required bool `json:"required"` | |
| 31 | + | State RequirementState `json:"state,omitempty"` | |
| 32 | + | TargetTurn int `json:"target_turn,omitempty"` | |
| 33 | + | ResultTurnID string `json:"result_turn_id,omitempty"` | |
| 34 | + | UpdatedAt time.Time `json:"updated_at,omitempty"` | |
| 35 | + | ErrorCode string `json:"error_code,omitempty"` | |
| 36 | + | Notified bool `json:"notified,omitempty"` | |
| 35 | 37 | } | |
| 36 | 38 | ||
| 37 | 39 | // Unmet reports whether this requirement still prevents parent completion. | |
| @@ -101,14 +103,22 @@ func (a *Agent) restoreRequirement(previous RequirementSnapshot) { | |||
| 101 | 103 | a.lifecycleMu.Unlock() | |
| 102 | 104 | } | |
| 103 | 105 | ||
| 104 | - | func (a *Agent) markRequirementNotified() { | |
| 106 | + | func (a *Agent) markRequirementNotified(expectedTurnID string) (bool, error) { | |
| 105 | 107 | if a == nil { | |
| 106 | - | return | |
| 108 | + | return false, nil | |
| 107 | 109 | } | |
| 108 | 110 | a.lifecycleMu.Lock() | |
| 111 | + | currentTurnID := strings.TrimSpace(a.requirement.ResultTurnID) | |
| 112 | + | if currentTurnID == "" && a.requirement.TargetTurn > 0 { | |
| 113 | + | currentTurnID = fmt.Sprintf("turn-%d", a.requirement.TargetTurn) | |
| 114 | + | } | |
| 115 | + | if expectedTurnID == "" || currentTurnID != expectedTurnID || a.result == nil || strings.TrimSpace(a.result.TurnID) != expectedTurnID || strings.TrimSpace(a.resultRef) == "" { | |
| 116 | + | a.lifecycleMu.Unlock() | |
| 117 | + | return false, fmt.Errorf("subagents: durable required result no longer matches %s", expectedTurnID) | |
| 118 | + | } | |
| 109 | 119 | if !a.requirement.Required || a.requirement.Notified { | |
| 110 | 120 | a.lifecycleMu.Unlock() | |
| 111 | - | return | |
| 121 | + | return false, nil | |
| 112 | 122 | } | |
| 113 | 123 | previous := a.requirement | |
| 114 | 124 | a.requirement.Notified = true | |
| @@ -123,7 +133,7 @@ func (a *Agent) markRequirementNotified() { | |||
| 123 | 133 | } | |
| 124 | 134 | a.lifecycleMu.Unlock() | |
| 125 | 135 | a.recordPersistenceError(err) | |
| 126 | - | return | |
| 136 | + | return false, err | |
| 127 | 137 | } | |
| 128 | 138 | } | |
| 129 | 139 | a.lifecycleMu.Lock() | |
| @@ -131,6 +141,7 @@ func (a *Agent) markRequirementNotified() { | |||
| 131 | 141 | a.signalRequirementLocked() | |
| 132 | 142 | } | |
| 133 | 143 | a.lifecycleMu.Unlock() | |
| 144 | + | return true, nil | |
| 134 | 145 | } | |
| 135 | 146 | ||
| 136 | 147 | func (a *Agent) resolveRequirement(step int, result *TurnResult, errMsg string, force bool) RequirementSnapshot { | |
| @@ -180,6 +191,9 @@ func (a *Agent) resolveRequirement(step int, result *TurnResult, errMsg string, | |||
| 180 | 191 | state, errorCode := classifyRequirementOutcome(result, errMsg) | |
| 181 | 192 | a.requirement.State = state | |
| 182 | 193 | a.requirement.ErrorCode = errorCode | |
| 194 | + | if result != nil { | |
| 195 | + | a.requirement.ResultTurnID = strings.TrimSpace(result.TurnID) | |
| 196 | + | } | |
| 183 | 197 | a.requirement.Notified = false | |
| 184 | 198 | a.requirement.UpdatedAt = time.Now().UTC() | |
| 185 | 199 | resolved := a.requirement | |
| @@ -309,7 +323,27 @@ func (f *Supervisor) MarkRequirementNotified(id string) error { | |||
| 309 | 323 | if a == nil { | |
| 310 | 324 | return fmt.Errorf("subagents: no such agent %q", id) | |
| 311 | 325 | } | |
| 312 | - | a.markRequirementNotified() | |
| 326 | + | snapshot := a.Snapshot() | |
| 327 | + | if strings.TrimSpace(snapshot.ResultRef) == "" { | |
| 328 | + | return errors.New("subagents: required result is not durably available") | |
| 329 | + | } | |
| 330 | + | requirement := snapshot.Requirement | |
| 331 | + | expectedTurnID := strings.TrimSpace(requirement.ResultTurnID) | |
| 332 | + | if expectedTurnID == "" && requirement.TargetTurn > 0 { | |
| 333 | + | expectedTurnID = fmt.Sprintf("turn-%d", requirement.TargetTurn) | |
| 334 | + | } | |
| 335 | + | result := snapshot.Result | |
| 336 | + | if result == nil || strings.TrimSpace(result.TurnID) != expectedTurnID { | |
| 337 | + | return fmt.Errorf("subagents: durable required result does not match %s", expectedTurnID) | |
| 338 | + | } | |
| 339 | + | marked, err := a.markRequirementNotified(expectedTurnID) | |
| 340 | + | if err != nil { | |
| 341 | + | a.recordTrace(TraceEvent{Type: "result.delivery.failed", TurnID: expectedTurnID, Data: map[string]any{"ref": ResultRef(a.ID)}}) | |
| 342 | + | return fmt.Errorf("subagents: persist required result delivery: %w", err) | |
| 343 | + | } | |
| 344 | + | if marked { | |
| 345 | + | a.recordTrace(TraceEvent{Type: "result.delivered", TurnID: expectedTurnID, Data: map[string]any{"ref": ResultRef(a.ID)}}) | |
| 346 | + | } | |
| 313 | 347 | return nil | |
| 314 | 348 | } | |
| 315 | 349 | ||
M packages/agent/subagents/requirement_test.go +94 -0
| @@ -5,6 +5,7 @@ import ( | |||
| 5 | 5 | "errors" | |
| 6 | 6 | "path/filepath" | |
| 7 | 7 | "runtime" | |
| 8 | + | "strings" | |
| 8 | 9 | "testing" | |
| 9 | 10 | "time" | |
| 10 | 11 | ) | |
| @@ -218,6 +219,99 @@ func TestRequiredOutcomePersistsAcrossReload(t *testing.T) { | |||
| 218 | 219 | } | |
| 219 | 220 | } | |
| 220 | 221 | ||
| 222 | + | func TestRequirementNotificationRecordsDeliveryExactlyOnce(t *testing.T) { | |
| 223 | + | trace := NewMemoryTraceWriter() | |
| 224 | + | t.Cleanup(func() { _ = trace.Close() }) | |
| 225 | + | a := &Agent{ID: "agent-1", trace: trace, result: &TurnResult{AgentID: "agent-1", TurnID: "turn-1", Status: ResultSucceeded}, resultRef: ResultRef("agent-1"), requirement: RequirementSnapshot{Required: true, State: RequirementSatisfied, TargetTurn: 1, ResultTurnID: "turn-1"}} | |
| 226 | + | s := &Supervisor{agents: map[string]*Agent{a.ID: a}} | |
| 227 | + | if err := s.MarkRequirementNotified(a.ID); err != nil { | |
| 228 | + | t.Fatal(err) | |
| 229 | + | } | |
| 230 | + | if err := s.MarkRequirementNotified(a.ID); err != nil { | |
| 231 | + | t.Fatal(err) | |
| 232 | + | } | |
| 233 | + | if err := trace.Flush(); err != nil { | |
| 234 | + | t.Fatal(err) | |
| 235 | + | } | |
| 236 | + | var delivered int | |
| 237 | + | for _, event := range trace.Events() { | |
| 238 | + | if event.Type == "result.delivered" { | |
| 239 | + | delivered++ | |
| 240 | + | } | |
| 241 | + | } | |
| 242 | + | if delivered != 1 { | |
| 243 | + | t.Fatalf("result.delivered count = %d, want 1", delivered) | |
| 244 | + | } | |
| 245 | + | } | |
| 246 | + | ||
| 247 | + | func TestRequirementNotificationRejectsResultSupersededByNewRequiredTurn(t *testing.T) { | |
| 248 | + | trace := NewMemoryTraceWriter() | |
| 249 | + | t.Cleanup(func() { _ = trace.Close() }) | |
| 250 | + | a := &Agent{ | |
| 251 | + | ID: "agent-1", | |
| 252 | + | trace: trace, | |
| 253 | + | result: &TurnResult{AgentID: "agent-1", TurnID: "turn-1", Status: ResultSucceeded}, | |
| 254 | + | resultRef: ResultRef("agent-1"), | |
| 255 | + | requirement: RequirementSnapshot{Required: true, State: RequirementSatisfied, TargetTurn: 1, ResultTurnID: "turn-1"}, | |
| 256 | + | } | |
| 257 | + | // A new required run begins after the caller observed turn-1 but before it | |
| 258 | + | // can publish delivery. The guarded transition must reject that stale turn. | |
| 259 | + | a.prepareRequired(2) | |
| 260 | + | if _, err := a.markRequirementNotified("turn-1"); err == nil { | |
| 261 | + | t.Fatal("stale result delivery was accepted") | |
| 262 | + | } | |
| 263 | + | if got := a.requirementSnapshot(); got.Notified || got.TargetTurn != 2 { | |
| 264 | + | t.Fatalf("requirement = %+v, want unnotified target turn 2", got) | |
| 265 | + | } | |
| 266 | + | } | |
| 267 | + | ||
| 268 | + | func TestRequirementNotificationPersistenceFailureIsReported(t *testing.T) { | |
| 269 | + | trace := NewMemoryTraceWriter() | |
| 270 | + | t.Cleanup(func() { _ = trace.Close() }) | |
| 271 | + | a := &Agent{ | |
| 272 | + | ID: "agent-1", | |
| 273 | + | trace: trace, | |
| 274 | + | result: &TurnResult{AgentID: "agent-1", TurnID: "turn-1", Status: ResultSucceeded}, | |
| 275 | + | resultRef: ResultRef("agent-1"), | |
| 276 | + | requirement: RequirementSnapshot{ | |
| 277 | + | Required: true, State: RequirementSatisfied, TargetTurn: 1, ResultTurnID: "turn-1", | |
| 278 | + | }, | |
| 279 | + | persistFn: func(*Agent) error { return errors.New("disk full") }, | |
| 280 | + | } | |
| 281 | + | s := &Supervisor{agents: map[string]*Agent{a.ID: a}} | |
| 282 | + | if err := s.MarkRequirementNotified(a.ID); err == nil || !strings.Contains(err.Error(), "disk full") { | |
| 283 | + | t.Fatalf("MarkRequirementNotified error = %v, want persistence failure", err) | |
| 284 | + | } | |
| 285 | + | if a.Snapshot().Requirement.Notified { | |
| 286 | + | t.Fatal("requirement remained notified after persistence failure") | |
| 287 | + | } | |
| 288 | + | if err := trace.Flush(); err != nil { | |
| 289 | + | t.Fatal(err) | |
| 290 | + | } | |
| 291 | + | view := trace.Views()[a.ID] | |
| 292 | + | if view.Result == nil || !view.Result.Failed { | |
| 293 | + | t.Fatalf("delivery failure trace = %#v", view.Result) | |
| 294 | + | } | |
| 295 | + | } | |
| 296 | + | ||
| 297 | + | func TestRequirementNotificationRejectsMissingDurableResult(t *testing.T) { | |
| 298 | + | trace := NewMemoryTraceWriter() | |
| 299 | + | t.Cleanup(func() { _ = trace.Close() }) | |
| 300 | + | a := &Agent{ID: "agent-1", trace: trace, requirement: RequirementSnapshot{Required: true, State: RequirementSatisfied, TargetTurn: 1}} | |
| 301 | + | s := &Supervisor{agents: map[string]*Agent{a.ID: a}} | |
| 302 | + | if err := s.MarkRequirementNotified(a.ID); err == nil { | |
| 303 | + | t.Fatal("MarkRequirementNotified succeeded without durable result") | |
| 304 | + | } | |
| 305 | + | if a.Snapshot().Requirement.Notified { | |
| 306 | + | t.Fatal("requirement marked notified without durable result") | |
| 307 | + | } | |
| 308 | + | for _, event := range trace.Events() { | |
| 309 | + | if event.Type == "result.delivered" { | |
| 310 | + | t.Fatalf("unexpected delivery trace: %#v", event) | |
| 311 | + | } | |
| 312 | + | } | |
| 313 | + | } | |
| 314 | + | ||
| 221 | 315 | func TestRequirementNotificationPersistsAndRetryClearsIt(t *testing.T) { | |
| 222 | 316 | root := t.TempDir() | |
| 223 | 317 | first := New(Config{ | |
M packages/agent/subagents/result.go +6 -1
| @@ -571,16 +571,21 @@ func (f *Supervisor) ensureResult(a *Agent, status Status, runErr error) { | |||
| 571 | 571 | result.Summary = strings.Split(strings.TrimSpace(result.Output), "\n")[0] | |
| 572 | 572 | } | |
| 573 | 573 | result = result.Bounded(a.maxOutputBytes, a.maxOutputLines) | |
| 574 | - | a.setResult(result) | |
| 575 | 574 | stateDir := a.stateDirectory(f.cfg.Root) | |
| 576 | 575 | if err := writeTurnResult(stateDir, result); err != nil { | |
| 577 | 576 | a.recordPersistenceError(fmt.Errorf("write turn result: %w", err)) | |
| 577 | + | failed := cloneTurnResult(result) | |
| 578 | + | failed.Status = ResultFailed | |
| 579 | + | failed.Error = &ResultError{Code: "result_persistence_failed", Message: "failed to persist delegated turn result"} | |
| 580 | + | a.setResult(failed) | |
| 578 | 581 | a.resolveRequirement(0, nil, fmt.Sprintf("write turn result: %v", err), true) | |
| 579 | 582 | return | |
| 580 | 583 | } | |
| 584 | + | a.setResult(result) | |
| 581 | 585 | a.lifecycleMu.Lock() | |
| 582 | 586 | a.resultRef = ResultRef(a.ID) | |
| 583 | 587 | a.lifecycleMu.Unlock() | |
| 588 | + | a.recordTrace(TraceEvent{Type: "result.available", TurnID: result.TurnID, Data: map[string]any{"ref": ResultRef(a.ID)}}) | |
| 584 | 589 | a.resolveRequirement(0, result, "", true) | |
| 585 | 590 | } | |
| 586 | 591 | ||
M packages/agent/subagents/result_test.go +35 -0
| @@ -182,6 +182,41 @@ func TestWriteTurnResultCleansTemporaryFileAfterRenameFailure(t *testing.T) { | |||
| 182 | 182 | } | |
| 183 | 183 | } | |
| 184 | 184 | ||
| 185 | + | func TestEnsureResultRecordsAvailabilityAfterDurableWrite(t *testing.T) { | |
| 186 | + | state := t.TempDir() | |
| 187 | + | trace := NewMemoryTraceWriter() | |
| 188 | + | t.Cleanup(func() { _ = trace.Close() }) | |
| 189 | + | agent := &Agent{ID: "agent-1", stateDir: state, trace: trace, maxOutputBytes: 1024, maxOutputLines: 10} | |
| 190 | + | (&Supervisor{}).ensureResult(agent, StatusDone, nil) | |
| 191 | + | if err := trace.Flush(); err != nil { | |
| 192 | + | t.Fatal(err) | |
| 193 | + | } | |
| 194 | + | view := ProjectTrace(trace.Events())[agent.ID] | |
| 195 | + | if view.Result == nil || !view.Result.Available || view.Result.Ref != ResultRef(agent.ID) { | |
| 196 | + | t.Fatalf("result trace = %#v", view.Result) | |
| 197 | + | } | |
| 198 | + | if _, err := os.Stat(resultPath(state)); err != nil { | |
| 199 | + | t.Fatalf("durable result missing: %v", err) | |
| 200 | + | } | |
| 201 | + | ||
| 202 | + | failedState := t.TempDir() | |
| 203 | + | if err := os.Mkdir(resultPath(failedState), 0o700); err != nil { | |
| 204 | + | t.Fatal(err) | |
| 205 | + | } | |
| 206 | + | failedTrace := NewMemoryTraceWriter() | |
| 207 | + | t.Cleanup(func() { _ = failedTrace.Close() }) | |
| 208 | + | failedAgent := &Agent{ID: "agent-2", stateDir: failedState, trace: failedTrace, maxOutputBytes: 1024, maxOutputLines: 10} | |
| 209 | + | (&Supervisor{}).ensureResult(failedAgent, StatusDone, nil) | |
| 210 | + | if err := failedTrace.Flush(); err != nil { | |
| 211 | + | t.Fatal(err) | |
| 212 | + | } | |
| 213 | + | for _, event := range failedTrace.Events() { | |
| 214 | + | if event.Type == "result.available" { | |
| 215 | + | t.Fatalf("result.available recorded after failed write: %#v", event) | |
| 216 | + | } | |
| 217 | + | } | |
| 218 | + | } | |
| 219 | + | ||
| 185 | 220 | func TestCaptureWorkspaceFailureIsDurableAndPrivate(t *testing.T) { | |
| 186 | 221 | makeAgent := func(stateDir string, capture func() (WorkspaceCapture, error)) *Agent { | |
| 187 | 222 | return &Agent{ID: "agent-1", stateDir: stateDir, workspaceCapture: capture} | |
M packages/agent/subagents/runner.go +119 -29
| @@ -78,7 +78,6 @@ type subagentWorkerArgsOpts struct { | |||
| 78 | 78 | FastMode bool | |
| 79 | 79 | FastModeSet bool | |
| 80 | 80 | Subagent string | |
| 81 | - | MaxTurns int | |
| 82 | 81 | TurnTimeout time.Duration | |
| 83 | 82 | LifetimeTurns int | |
| 84 | 83 | RunTurns int | |
| @@ -117,7 +116,6 @@ func defaultChildArgs(exe string, a *Agent, sessionPath, inboxPath string) []str | |||
| 117 | 116 | FastMode: a.FastMode, | |
| 118 | 117 | FastModeSet: true, | |
| 119 | 118 | Subagent: a.Subagent, | |
| 120 | - | MaxTurns: a.MaxTurns, | |
| 121 | 119 | TurnTimeout: a.Timeout, | |
| 122 | 120 | LifetimeTurns: a.LifetimeTurnsValue(), | |
| 123 | 121 | RunTurns: a.CurrentRunTurnsValue(), | |
| @@ -162,9 +160,6 @@ func subagentWorkerArgs(opts subagentWorkerArgsOpts) []string { | |||
| 162 | 160 | if opts.Subagent != "" { | |
| 163 | 161 | args = append(args, "--subagent", opts.Subagent) | |
| 164 | 162 | } | |
| 165 | - | if opts.MaxTurns > 0 { | |
| 166 | - | args = append(args, "--max-turns", fmt.Sprint(opts.MaxTurns)) | |
| 167 | - | } | |
| 168 | 163 | if opts.TurnTimeout > 0 { | |
| 169 | 164 | args = append(args, "--subagent-turn-timeout", opts.TurnTimeout.String()) | |
| 170 | 165 | } | |
| @@ -708,6 +703,7 @@ func updateAgentFromEvent(a *Agent, ev Event) error { | |||
| 708 | 703 | a.markActivity(now) | |
| 709 | 704 | persist := false | |
| 710 | 705 | notifyIdle := false | |
| 706 | + | var eventErr error | |
| 711 | 707 | switch ev.Type { | |
| 712 | 708 | case EventAgentReady, "agent_ready": | |
| 713 | 709 | a.setProcessState(ProcessAlive) | |
| @@ -759,20 +755,27 @@ func updateAgentFromEvent(a *Agent, ev Event) error { | |||
| 759 | 755 | a.setProcessState(ProcessAlive) | |
| 760 | 756 | case EventTurnResult, "turn_result": | |
| 761 | 757 | if result, err := decodeTurnResultEvent(ev, a.ID, a.maxOutputBytes, a.maxOutputLines); err == nil { | |
| 762 | - | a.setResult(result) | |
| 763 | - | if a.turnResults != nil { | |
| 764 | - | select { | |
| 765 | - | case a.turnResults <- result: | |
| 766 | - | default: | |
| 767 | - | } | |
| 768 | - | } | |
| 769 | 758 | if a.stateDir != "" { | |
| 770 | - | if err := writeTurnResult(a.stateDir, result); err == nil { | |
| 759 | + | if err := writeTurnResult(a.stateDir, result); err != nil { | |
| 760 | + | eventErr = fmt.Errorf("write turn result: %w", err) | |
| 761 | + | result = cloneTurnResult(result) | |
| 762 | + | result.Status = ResultFailed | |
| 763 | + | result.Error = &ResultError{ | |
| 764 | + | Code: "result_persistence_failed", | |
| 765 | + | Message: "subagent result could not be persisted; inspect the live session before retrying", | |
| 766 | + | } | |
| 767 | + | a.lifecycleMu.Lock() | |
| 768 | + | a.resultRef = "" | |
| 769 | + | a.lifecycleMu.Unlock() | |
| 770 | + | a.recordPersistenceError(eventErr) | |
| 771 | + | } else { | |
| 771 | 772 | a.lifecycleMu.Lock() | |
| 772 | 773 | a.resultRef = ResultRef(a.ID) | |
| 773 | 774 | a.lifecycleMu.Unlock() | |
| 775 | + | a.recordTrace(TraceEvent{Type: "result.available", TurnID: result.TurnID, Data: map[string]any{"ref": ResultRef(a.ID)}}) | |
| 774 | 776 | } | |
| 775 | 777 | } | |
| 778 | + | a.setResult(result) | |
| 776 | 779 | switch result.Status { | |
| 777 | 780 | case ResultCanceled: | |
| 778 | 781 | a.setTurnState(TurnCanceled, result.TurnID) | |
| @@ -801,13 +804,8 @@ func updateAgentFromEvent(a *Agent, ev Event) error { | |||
| 801 | 804 | case "error": | |
| 802 | 805 | if code, _ := ev.Data["code"].(string); code == "turn_rejected" && eventMatchesPendingResume(a, ev) { | |
| 803 | 806 | if a.rejectActiveResumePrompt() { | |
| 804 | - | reason, _ := ev.Data["reason"].(string) | |
| 805 | - | if reason == "max_turns" { | |
| 806 | - | a.setTurnState(TurnFailed, ev.TurnID) | |
| 807 | - | } else { | |
| 808 | - | a.setTurnState(TurnIdle, ev.TurnID) | |
| 809 | - | notifyIdle = true | |
| 810 | - | } | |
| 807 | + | a.setTurnState(TurnIdle, ev.TurnID) | |
| 808 | + | notifyIdle = true | |
| 811 | 809 | persist = true | |
| 812 | 810 | } | |
| 813 | 811 | } | |
| @@ -820,7 +818,9 @@ func updateAgentFromEvent(a *Agent, ev Event) error { | |||
| 820 | 818 | break | |
| 821 | 819 | } | |
| 822 | 820 | message, _ := ev.Data["error"].(string) | |
| 823 | - | if message != "" { | |
| 821 | + | if result := a.Result(); result != nil && result.Error != nil && result.Error.Code == "result_persistence_failed" { | |
| 822 | + | a.setTurnState(TurnFailed, ev.TurnID) | |
| 823 | + | } else if message != "" { | |
| 824 | 824 | a.setTurnState(TurnFailed, ev.TurnID) | |
| 825 | 825 | } else { | |
| 826 | 826 | a.setTurnState(TurnSucceeded, ev.TurnID) | |
| @@ -850,13 +850,13 @@ func updateAgentFromEvent(a *Agent, ev Event) error { | |||
| 850 | 850 | if persist && a.persistFn != nil { | |
| 851 | 851 | if err := a.persistFn(a); err != nil { | |
| 852 | 852 | a.recordPersistenceError(err) | |
| 853 | - | return err | |
| 853 | + | eventErr = errors.Join(eventErr, err) | |
| 854 | 854 | } | |
| 855 | 855 | } | |
| 856 | 856 | if notifyIdle { | |
| 857 | 857 | a.notifyTurnIdle() | |
| 858 | 858 | } | |
| 859 | - | return nil | |
| 859 | + | return eventErr | |
| 860 | 860 | } | |
| 861 | 861 | ||
| 862 | 862 | // notifyPromptTurnEnd calls Agent.OnTurnEnd only for the subagent | |
| @@ -903,37 +903,127 @@ func recordWorkerTrace(agent *Agent, ev Event) { | |||
| 903 | 903 | if agent == nil { | |
| 904 | 904 | return | |
| 905 | 905 | } | |
| 906 | + | // assistant_start only marks the beginning of a provider response. It is | |
| 907 | + | // neither streamed content nor an open operation, and recording it as an | |
| 908 | + | // unknown protocol observation makes it replace useful live activity. | |
| 909 | + | if ev.Type == "assistant_start" || ev.Type == EventAgentHeartbeat || ev.Type == "agent_heartbeat" { | |
| 910 | + | return | |
| 911 | + | } | |
| 906 | 912 | typeName := strings.ReplaceAll(ev.Type, "_", ".") | |
| 907 | - | traceType := "worker.event" | |
| 913 | + | traceType := "worker.protocol.observed" | |
| 914 | + | nestedTurn, _ := ev.Data["nested_turn"].(bool) | |
| 908 | 915 | switch ev.Type { | |
| 909 | 916 | case "turn_start", EventTurnStarted: | |
| 910 | - | traceType = "turn.started" | |
| 911 | - | case "turn_end", EventTurnResult: | |
| 917 | + | if nestedTurn { | |
| 918 | + | traceType = "provider.request.started" | |
| 919 | + | } else { | |
| 920 | + | traceType = "turn.started" | |
| 921 | + | } | |
| 922 | + | case "turn_end": | |
| 923 | + | // Each delegated turn has completed its provider/model loop by this | |
| 924 | + | // point. Close the provider operation even when the terminal result was | |
| 925 | + | // emitted through a separate turn.result event. | |
| 926 | + | if errMsg, _ := ev.Data["error"].(string); errMsg != "" { | |
| 927 | + | traceType = "provider.request.failed" | |
| 928 | + | } else { | |
| 929 | + | traceType = "provider.request.finished" | |
| 930 | + | } | |
| 931 | + | case EventTurnResult: | |
| 912 | 932 | traceType = "turn.finished" | |
| 933 | + | if status, _ := ev.Data["status"].(string); status == "failed" { | |
| 934 | + | traceType = "turn.failed" | |
| 935 | + | } else if status == "canceled" { | |
| 936 | + | traceType = "turn.cancelled" | |
| 937 | + | } | |
| 913 | 938 | case "turn_failed", EventTurnFailed: | |
| 914 | 939 | traceType = "turn.failed" | |
| 915 | - | case "tool_call", EventToolStarted: | |
| 940 | + | case "turn_cancelled", "turn.canceled": | |
| 941 | + | traceType = "turn.cancelled" | |
| 942 | + | case "request_started": | |
| 943 | + | traceType = "provider.request.started" | |
| 944 | + | case "retry_scheduled": | |
| 945 | + | traceType = "provider.request.retry.scheduled" | |
| 946 | + | case "error": | |
| 947 | + | // Error events do not necessarily belong to a provider request; the | |
| 948 | + | // following terminal turn event closes any request factually. | |
| 949 | + | case "tool_execution_started", EventToolStarted: | |
| 916 | 950 | traceType = "tool.started" | |
| 917 | 951 | case "tool_result", EventToolFinished: | |
| 918 | 952 | traceType = "tool.finished" | |
| 953 | + | case EventMessageDelta: | |
| 954 | + | traceType = "assistant.stream.observed" | |
| 919 | 955 | case "agent_ready", EventAgentReady: | |
| 920 | 956 | traceType = "agent.ready" | |
| 921 | 957 | case "agent_stopped": | |
| 922 | 958 | traceType = "agent.stopped" | |
| 923 | 959 | } | |
| 924 | - | data := map[string]any{"worker_event": typeName} | |
| 960 | + | data := map[string]any{"source_event": typeName} | |
| 925 | 961 | if name, ok := ev.Data["name"].(string); ok && name != "" { | |
| 926 | 962 | data["name"] = name | |
| 927 | 963 | } | |
| 928 | 964 | if status, ok := ev.Data["status"].(string); ok && status != "" { | |
| 929 | 965 | data["status"] = status | |
| 930 | 966 | } | |
| 931 | - | if callID, ok := ev.Data["call_id"].(string); ok && callID != "" { | |
| 967 | + | for _, key := range []string{"provider", "model"} { | |
| 968 | + | if value, ok := ev.Data[key].(string); ok && value != "" { | |
| 969 | + | data[key] = value | |
| 970 | + | } | |
| 971 | + | } | |
| 972 | + | for _, key := range []string{"step", "attempt", "max_attempts"} { | |
| 973 | + | if value, ok := ev.Data[key].(float64); ok && value > 0 { | |
| 974 | + | data[key] = int(value) | |
| 975 | + | } else if value, ok := ev.Data[key].(int); ok && value > 0 { | |
| 976 | + | data[key] = value | |
| 977 | + | } | |
| 978 | + | } | |
| 979 | + | if errorCode := traceErrorCode(ev.Data); errorCode != "" { | |
| 980 | + | data["error_code"] = errorCode | |
| 981 | + | } | |
| 982 | + | callID, _ := ev.Data["call_id"].(string) | |
| 983 | + | if callID == "" { | |
| 984 | + | callID, _ = ev.Data["id"].(string) | |
| 985 | + | } | |
| 986 | + | if callID != "" { | |
| 932 | 987 | data["call_id"] = callID | |
| 933 | 988 | } | |
| 934 | 989 | agent.recordTrace(TraceEvent{Type: traceType, Timestamp: ev.Time, TurnID: ev.TurnID, Data: data}) | |
| 935 | 990 | } | |
| 936 | 991 | ||
| 992 | + | func traceErrorCode(data map[string]any) string { | |
| 993 | + | if structured, ok := data["error"].(map[string]any); ok { | |
| 994 | + | if code, _ := structured["code"].(string); isTraceErrorCode(code) { | |
| 995 | + | return code | |
| 996 | + | } | |
| 997 | + | } | |
| 998 | + | value, _ := data["error"].(string) | |
| 999 | + | value = strings.ToLower(value) | |
| 1000 | + | switch { | |
| 1001 | + | case strings.Contains(value, "deadline") || strings.Contains(value, "timeout"): | |
| 1002 | + | return "deadline_exceeded" | |
| 1003 | + | case strings.Contains(value, "canceled") || strings.Contains(value, "cancelled"): | |
| 1004 | + | return "cancelled" | |
| 1005 | + | case strings.Contains(value, "quota") || strings.Contains(value, "billing"): | |
| 1006 | + | return "quota" | |
| 1007 | + | case strings.Contains(value, "rate limit") || strings.Contains(value, "429"): | |
| 1008 | + | return "rate_limit" | |
| 1009 | + | case strings.Contains(value, "context") && strings.Contains(value, "limit"): | |
| 1010 | + | return "context_limit" | |
| 1011 | + | case value != "": | |
| 1012 | + | return "provider_error" | |
| 1013 | + | default: | |
| 1014 | + | return "" | |
| 1015 | + | } | |
| 1016 | + | } | |
| 1017 | + | ||
| 1018 | + | func isTraceErrorCode(code string) bool { | |
| 1019 | + | switch code { | |
| 1020 | + | case "deadline_exceeded", "stream_idle_timeout", "cancelled", "quota", "rate_limit", "context_limit", "provider_error": | |
| 1021 | + | return true | |
| 1022 | + | default: | |
| 1023 | + | return false | |
| 1024 | + | } | |
| 1025 | + | } | |
| 1026 | + | ||
| 937 | 1027 | // applyEventToSink translates an Event into Sink updates. Only a | |
| 938 | 1028 | // few event types are interpreted; the rest still land in the | |
| 939 | 1029 | // durable log via the caller. | |
M packages/agent/subagents/runner_test.go +184 -0
| @@ -360,6 +360,15 @@ func TestDefaultChildArgsPropagatesWebSearchPolicy(t *testing.T) { | |||
| 360 | 360 | } | |
| 361 | 361 | } | |
| 362 | 362 | ||
| 363 | + | func TestDefaultChildArgsDoNotImposeModelStepOrTurnLimits(t *testing.T) { | |
| 364 | + | args := defaultChildArgs("zut", &Agent{Task: "task"}, "/session", "/inbox") | |
| 365 | + | for _, flag := range []string{"--max-steps", "--max-turns"} { | |
| 366 | + | if indexOf(args, flag) >= 0 { | |
| 367 | + | t.Fatalf("argv = %v, contains removed subagent limit %s", args, flag) | |
| 368 | + | } | |
| 369 | + | } | |
| 370 | + | } | |
| 371 | + | ||
| 363 | 372 | func TestResolveSubagentExecutableUsesCurrentWhenAvailable(t *testing.T) { | |
| 364 | 373 | got, err := resolveSubagentExecutable(context.Background(), "/current/bin/zut", "/old/bin/zut", func(_ context.Context, candidate string) (string, error) { | |
| 365 | 374 | if candidate == "/current/bin/zut" { | |
| @@ -754,6 +763,64 @@ func TestRecordPersistenceErrorPreservesFirstFailure(t *testing.T) { | |||
| 754 | 763 | } | |
| 755 | 764 | } | |
| 756 | 765 | ||
| 766 | + | func TestUpdateAgentFromEventSurfacesTurnResultPersistenceFailure(t *testing.T) { | |
| 767 | + | blockedStateDir := filepath.Join(t.TempDir(), "not-a-directory") | |
| 768 | + | if err := os.WriteFile(blockedStateDir, []byte("blocked"), 0o600); err != nil { | |
| 769 | + | t.Fatal(err) | |
| 770 | + | } | |
| 771 | + | a := &Agent{ID: "agent-1", stateDir: blockedStateDir} | |
| 772 | + | err := updateAgentFromEvent(a, Event{Type: EventTurnResult, TurnID: "turn-1", Data: map[string]any{ | |
| 773 | + | "status": "succeeded", | |
| 774 | + | "output": "review complete", | |
| 775 | + | }}) | |
| 776 | + | if err == nil { | |
| 777 | + | t.Fatal("turn-result persistence failure was ignored") | |
| 778 | + | } | |
| 779 | + | result := a.Result() | |
| 780 | + | if result == nil || result.Status != ResultFailed || result.Error == nil || result.Error.Code != "result_persistence_failed" { | |
| 781 | + | t.Fatalf("in-memory result = %#v, want explicit persistence failure", result) | |
| 782 | + | } | |
| 783 | + | if result.Output != "review complete" { | |
| 784 | + | t.Fatalf("failure output = %q, want preserved partial output", result.Output) | |
| 785 | + | } | |
| 786 | + | } | |
| 787 | + | ||
| 788 | + | func TestUpdateAgentFromEventRetainsPersistenceFailureAfterTurnEnd(t *testing.T) { | |
| 789 | + | blockedStateDir := filepath.Join(t.TempDir(), "not-a-directory") | |
| 790 | + | if err := os.WriteFile(blockedStateDir, []byte("blocked"), 0o600); err != nil { | |
| 791 | + | t.Fatal(err) | |
| 792 | + | } | |
| 793 | + | a := &Agent{ID: "agent-1", stateDir: blockedStateDir} | |
| 794 | + | if err := updateAgentFromEvent(a, Event{Type: EventTurnResult, TurnID: "turn-1", Data: map[string]any{"status": "succeeded"}}); err == nil { | |
| 795 | + | t.Fatal("turn-result persistence failure was ignored") | |
| 796 | + | } | |
| 797 | + | if err := updateAgentFromEvent(a, Event{Type: "turn_end", TurnID: "turn-1", Data: map[string]any{"step": 1}}); err != nil { | |
| 798 | + | t.Fatal(err) | |
| 799 | + | } | |
| 800 | + | if got := a.TurnState(); got != TurnFailed { | |
| 801 | + | t.Fatalf("turn state = %q, want %q", got, TurnFailed) | |
| 802 | + | } | |
| 803 | + | } | |
| 804 | + | ||
| 805 | + | func TestUpdateAgentFromEventRecordsDurableResultAvailability(t *testing.T) { | |
| 806 | + | trace := NewMemoryTraceWriter() | |
| 807 | + | t.Cleanup(func() { _ = trace.Close() }) | |
| 808 | + | a := &Agent{ID: "agent-1", stateDir: t.TempDir(), trace: trace} | |
| 809 | + | if err := updateAgentFromEvent(a, Event{Type: EventTurnResult, TurnID: "turn-1", Data: map[string]any{ | |
| 810 | + | "status": "succeeded", | |
| 811 | + | "output": "review complete", | |
| 812 | + | }}); err != nil { | |
| 813 | + | t.Fatal(err) | |
| 814 | + | } | |
| 815 | + | if err := trace.Flush(); err != nil { | |
| 816 | + | t.Fatal(err) | |
| 817 | + | } | |
| 818 | + | view := trace.Views()[a.ID] | |
| 819 | + | if view.Result == nil || !view.Result.Available || view.Result.Ref != ResultRef(a.ID) { | |
| 820 | + | t.Fatalf("live result trace = %#v, want durable availability", view.Result) | |
| 821 | + | } | |
| 822 | + | } | |
| 823 | + | ||
| 757 | 824 | func TestRestoreResumePromptPreservesCommandIdentity(t *testing.T) { | |
| 758 | 825 | a := &Agent{} | |
| 759 | 826 | a.setResumePrompt("old prompt", time.Now()) | |
| @@ -801,6 +868,123 @@ func TestEventCounterRejectsNegativeFractionalAndOverflowValues(t *testing.T) { | |||
| 801 | 868 | } | |
| 802 | 869 | } | |
| 803 | 870 | ||
| 871 | + | func TestRecordWorkerTraceNormalizesSafeLiveObservations(t *testing.T) { | |
| 872 | + | trace := NewMemoryTraceWriter() | |
| 873 | + | t.Cleanup(func() { _ = trace.Close() }) | |
| 874 | + | agent := &Agent{ID: "agent-1", trace: trace} | |
| 875 | + | recordWorkerTrace(agent, Event{Type: EventMessageDelta, TurnID: "turn-1", Data: map[string]any{"delta": "sensitive answer"}}) | |
| 876 | + | recordWorkerTrace(agent, Event{Type: "assistant_start", TurnID: "turn-1"}) | |
| 877 | + | recordWorkerTrace(agent, Event{Type: EventAgentHeartbeat, TurnID: "turn-1"}) | |
| 878 | + | recordWorkerTrace(agent, Event{Type: "request_started", TurnID: "turn-1", Data: map[string]any{"provider": "test", "model": "test"}}) | |
| 879 | + | // The worker canonicalizes core turn_start to EventTurnStarted while | |
| 880 | + | // retaining nested_turn, so this exercises the supervisor's real input. | |
| 881 | + | recordWorkerTrace(agent, Event{Type: EventTurnStarted, TurnID: "turn-1", Data: map[string]any{"nested_turn": true}}) | |
| 882 | + | recordWorkerTrace(agent, Event{Type: "turn_end", TurnID: "turn-1", Data: map[string]any{"nested_turn": true}}) | |
| 883 | + | recordWorkerTrace(agent, Event{Type: "tool_call", TurnID: "turn-1", Data: map[string]any{"id": "call-1", "name": "bash", "args": "sensitive arguments"}}) | |
| 884 | + | recordWorkerTrace(agent, Event{Type: "tool_execution_started", TurnID: "turn-1", Data: map[string]any{"id": "call-1", "name": "bash"}}) | |
| 885 | + | recordWorkerTrace(agent, Event{Type: "tool_use_args", TurnID: "turn-1", Data: map[string]any{"id": "call-1", "delta": "sensitive argument fragment"}}) | |
| 886 | + | recordWorkerTrace(agent, Event{Type: "future_event", TurnID: "turn-1", Data: map[string]any{"secret": "hidden"}}) | |
| 887 | + | if err := trace.Flush(); err != nil { | |
| 888 | + | t.Fatal(err) | |
| 889 | + | } | |
| 890 | + | events := trace.Events() | |
| 891 | + | if len(events) != 8 { | |
| 892 | + | t.Fatalf("events = %#v", events) | |
| 893 | + | } | |
| 894 | + | if got, want := events[0].Type, "assistant.stream.observed"; got != want { | |
| 895 | + | t.Fatalf("type = %q, want %q", got, want) | |
| 896 | + | } | |
| 897 | + | if _, found := events[0].Data["delta"]; found { | |
| 898 | + | t.Fatalf("stream payload leaked: %#v", events[0].Data) | |
| 899 | + | } | |
| 900 | + | if got, want := events[1].Type, "provider.request.started"; got != want { | |
| 901 | + | t.Fatalf("type = %q, want %q", got, want) | |
| 902 | + | } | |
| 903 | + | if got, want := events[2].Type, "provider.request.started"; got != want { | |
| 904 | + | t.Fatalf("type = %q, want %q", got, want) | |
| 905 | + | } | |
| 906 | + | if got, want := events[3].Type, "provider.request.finished"; got != want { | |
| 907 | + | t.Fatalf("type = %q, want %q", got, want) | |
| 908 | + | } | |
| 909 | + | if got, want := events[4].Type, "worker.protocol.observed"; got != want { | |
| 910 | + | t.Fatalf("tool_call type = %q, want %q", got, want) | |
| 911 | + | } | |
| 912 | + | if got, want := events[5].Type, "tool.started"; got != want { | |
| 913 | + | t.Fatalf("type = %q, want %q", got, want) | |
| 914 | + | } | |
| 915 | + | if got, _ := events[5].Data["call_id"].(string); got != "call-1" { | |
| 916 | + | t.Fatalf("tool call_id = %q", got) | |
| 917 | + | } | |
| 918 | + | if _, found := events[5].Data["args"]; found { | |
| 919 | + | t.Fatalf("tool arguments leaked: %#v", events[5].Data) | |
| 920 | + | } | |
| 921 | + | for _, index := range []int{6, 7} { | |
| 922 | + | if got, want := events[index].Type, "worker.protocol.observed"; got != want { | |
| 923 | + | t.Fatalf("events[%d].type = %q, want %q", index, got, want) | |
| 924 | + | } | |
| 925 | + | if _, found := events[index].Data["delta"]; found { | |
| 926 | + | t.Fatalf("worker payload leaked: %#v", events[index].Data) | |
| 927 | + | } | |
| 928 | + | if _, found := events[index].Data["secret"]; found { | |
| 929 | + | t.Fatalf("worker payload leaked: %#v", events[index].Data) | |
| 930 | + | } | |
| 931 | + | } | |
| 932 | + | view := ProjectTrace(append([]TraceEvent{{Type: "turn.started", AgentID: "agent-1", TurnID: "turn-1"}}, events...))["agent-1"] | |
| 933 | + | if primaryOperation(t, view) == nil || primaryOperation(t, view).Type != "tool.started" || primaryOperation(t, view).Name != "bash" { | |
| 934 | + | t.Fatalf("current-turn tool activity was not projected: %#v", view) | |
| 935 | + | } | |
| 936 | + | } | |
| 937 | + | ||
| 938 | + | func TestRecordWorkerTraceClosesProviderRequestWhenTurnEnds(t *testing.T) { | |
| 939 | + | trace := NewMemoryTraceWriter() | |
| 940 | + | t.Cleanup(func() { _ = trace.Close() }) | |
| 941 | + | agent := &Agent{ID: "agent-1", trace: trace} | |
| 942 | + | recordWorkerTrace(agent, Event{Type: "request_started", TurnID: "turn-1"}) | |
| 943 | + | recordWorkerTrace(agent, Event{Type: "turn_end", TurnID: "turn-1", Data: map[string]any{"error": "provider unavailable"}}) | |
| 944 | + | if err := trace.Flush(); err != nil { | |
| 945 | + | t.Fatal(err) | |
| 946 | + | } | |
| 947 | + | view := ProjectTrace(trace.Events())["agent-1"] | |
| 948 | + | if len(view.OpenOperations) != 0 { | |
| 949 | + | t.Fatalf("open operations = %#v, want none", view.OpenOperations) | |
| 950 | + | } | |
| 951 | + | if view.LastEvent.Type != "provider.request.failed" { | |
| 952 | + | t.Fatalf("last event = %#v, want provider request failure", view.LastEvent) | |
| 953 | + | } | |
| 954 | + | } | |
| 955 | + | ||
| 956 | + | func TestRecordWorkerTracePreservesStructuredErrorCode(t *testing.T) { | |
| 957 | + | trace := NewMemoryTraceWriter() | |
| 958 | + | t.Cleanup(func() { _ = trace.Close() }) | |
| 959 | + | agent := &Agent{ID: "agent-1", trace: trace} | |
| 960 | + | recordWorkerTrace(agent, Event{Type: EventTurnResult, TurnID: "turn-1", Data: map[string]any{"status": "failed", "error": map[string]any{"code": "stream_idle_timeout"}}}) | |
| 961 | + | if err := trace.Flush(); err != nil { | |
| 962 | + | t.Fatal(err) | |
| 963 | + | } | |
| 964 | + | events := trace.Events() | |
| 965 | + | if len(events) != 1 || events[0].Data["error_code"] != "stream_idle_timeout" { | |
| 966 | + | t.Fatalf("trace events = %#v", events) | |
| 967 | + | } | |
| 968 | + | } | |
| 969 | + | ||
| 970 | + | func TestRecordWorkerTraceProjectsCurrentModelStep(t *testing.T) { | |
| 971 | + | trace := NewMemoryTraceWriter() | |
| 972 | + | t.Cleanup(func() { _ = trace.Close() }) | |
| 973 | + | agent := &Agent{ID: "agent-1", trace: trace} | |
| 974 | + | recordWorkerTrace(agent, Event{ | |
| 975 | + | Type: EventTurnStarted, | |
| 976 | + | TurnID: "turn-1", | |
| 977 | + | Data: map[string]any{"nested_turn": true, "step": float64(3)}, | |
| 978 | + | }) | |
| 979 | + | if err := trace.Flush(); err != nil { | |
| 980 | + | t.Fatal(err) | |
| 981 | + | } | |
| 982 | + | operation := primaryOperation(t, ProjectTrace(trace.Events())["agent-1"]) | |
| 983 | + | if operation.Type != "provider.request.started" || operation.Step != 3 { | |
| 984 | + | t.Fatalf("provider operation = %#v, want model step 3", operation) | |
| 985 | + | } | |
| 986 | + | } | |
| 987 | + | ||
| 804 | 988 | func TestApplyEventToSinkProjectsToolActivityIntoTranscript(t *testing.T) { | |
| 805 | 989 | events := []Event{ | |
| 806 | 990 | {Type: EventToolStarted, Data: map[string]any{"name": "bash"}}, | |
M packages/agent/subagents/scheduler.go +0 -3
| @@ -63,9 +63,6 @@ func (f *Supervisor) validateSpawnRequest(req SpawnRequest) error { | |||
| 63 | 63 | ||
| 64 | 64 | func (f *Supervisor) validateSpawnOptions(req SpawnRequest) error { | |
| 65 | 65 | p := f.cfg.Policy | |
| 66 | - | if req.MaxTurns < 0 || req.MaxTurns > p.MaxTurns { | |
| 67 | - | return fmt.Errorf("subagents: max_turns must be 0 through %d (zero uses the policy default)", p.MaxTurns) | |
| 68 | - | } | |
| 69 | 66 | if req.Timeout < 0 { | |
| 70 | 67 | return errorsInvalid("timeout must not be negative") | |
| 71 | 68 | } | |
M packages/agent/subagents/supervisor.go +1 -16
| @@ -228,14 +228,6 @@ func (f *Supervisor) RepoRoot() string { | |||
| 228 | 228 | return f.cfg.RepoRoot | |
| 229 | 229 | } | |
| 230 | 230 | ||
| 231 | - | // MaxTurns returns the maximum prompt-level turns allowed for subsequently | |
| 232 | - | // spawned agents. | |
| 233 | - | func (f *Supervisor) MaxTurns() int { | |
| 234 | - | f.mu.Lock() | |
| 235 | - | defer f.mu.Unlock() | |
| 236 | - | return f.cfg.Policy.MaxTurns | |
| 237 | - | } | |
| 238 | - | ||
| 239 | 231 | // SetFastMode updates the fast-mode setting for subsequently spawned | |
| 240 | 232 | // children. Existing agents keep the setting captured at their spawn | |
| 241 | 233 | // boundary so resumes remain deterministic. | |
| @@ -382,7 +374,6 @@ type SpawnRequest struct { | |||
| 382 | 374 | // supervisor persists its outcome while the manager remains asynchronous. | |
| 383 | 375 | Required bool | |
| 384 | 376 | Timeout time.Duration | |
| 385 | - | MaxTurns int | |
| 386 | 377 | Tools []string | |
| 387 | 378 | WorkspaceMode WorkspaceMode | |
| 388 | 379 | WorkspaceBase string | |
| @@ -502,10 +493,6 @@ func (f *Supervisor) SpawnReq(ctx context.Context, req SpawnRequest) (*Agent, er | |||
| 502 | 493 | return nil, err | |
| 503 | 494 | } | |
| 504 | 495 | dir = workspace.Dir() | |
| 505 | - | maxTurns := req.MaxTurns | |
| 506 | - | if maxTurns <= 0 { | |
| 507 | - | maxTurns = f.cfg.Policy.MaxTurns | |
| 508 | - | } | |
| 509 | 496 | timeout := req.Timeout | |
| 510 | 497 | if timeout <= 0 { | |
| 511 | 498 | timeout = f.cfg.Policy.DefaultTimeout | |
| @@ -539,7 +526,6 @@ func (f *Supervisor) SpawnReq(ctx context.Context, req SpawnRequest) (*Agent, er | |||
| 539 | 526 | WorkspaceMode: workspaceMode, | |
| 540 | 527 | WorkspaceBase: strings.TrimSpace(req.WorkspaceBase), | |
| 541 | 528 | WorkspaceCapture: req.WorkspaceCapture, | |
| 542 | - | MaxTurns: maxTurns, | |
| 543 | 529 | Timeout: timeout, | |
| 544 | 530 | HeartbeatInterval: f.cfg.Policy.HeartbeatInterval, | |
| 545 | 531 | Tools: tools, | |
| @@ -561,7 +547,6 @@ func (f *Supervisor) SpawnReq(ctx context.Context, req SpawnRequest) (*Agent, er | |||
| 561 | 547 | maxOutputBytes: f.cfg.Policy.MaxOutputBytes, | |
| 562 | 548 | maxOutputLines: f.cfg.Policy.MaxOutputLines, | |
| 563 | 549 | done: make(chan struct{}), | |
| 564 | - | turnResults: make(chan *TurnResult, 16), | |
| 565 | 550 | } | |
| 566 | 551 | a.fastModeOverridesHost = fastModeOverridesHost | |
| 567 | 552 | if a.RootSessionID == "" { | |
| @@ -643,7 +628,7 @@ func (f *Supervisor) TraceViews() map[string]AgentTraceView { | |||
| 643 | 628 | if f == nil || f.trace == nil { | |
| 644 | 629 | return map[string]AgentTraceView{} | |
| 645 | 630 | } | |
| 646 | - | return ProjectTrace(f.trace.Events()) | |
| 631 | + | return f.trace.Views() | |
| 647 | 632 | } | |
| 648 | 633 | ||
| 649 | 634 | // Close flushes and closes the optional trace bundle. It does not stop agents; | |
M packages/agent/subagents/supervisor_test.go +24 -0
| @@ -72,6 +72,30 @@ func TestSpawnTimeoutDoesNotLimitResumableWorkerLifetime(t *testing.T) { | |||
| 72 | 72 | a.Wait() | |
| 73 | 73 | } | |
| 74 | 74 | ||
| 75 | + | func TestSpawnWithoutTimeoutUsesUnlimitedTurns(t *testing.T) { | |
| 76 | + | f := newTestSupervisor(t, func(*Agent) Runner { | |
| 77 | + | return RunnerFunc(func(context.Context, Sink) error { return nil }) | |
| 78 | + | }) | |
| 79 | + | a, err := f.Spawn(context.Background(), "unlimited turn") | |
| 80 | + | if err != nil { | |
| 81 | + | t.Fatal(err) | |
| 82 | + | } | |
| 83 | + | if a.Timeout != 0 { | |
| 84 | + | t.Fatalf("turn timeout = %s, want unlimited", a.Timeout) | |
| 85 | + | } | |
| 86 | + | a.Wait() | |
| 87 | + | } | |
| 88 | + | ||
| 89 | + | func TestSubagentPolicyCanonicalizesNegativeUnlimitedTimeout(t *testing.T) { | |
| 90 | + | policy := SubagentPolicy{ | |
| 91 | + | DefaultTimeout: -time.Second, | |
| 92 | + | } | |
| 93 | + | policy.normalize() | |
| 94 | + | if policy.DefaultTimeout != 0 { | |
| 95 | + | t.Fatalf("normalized timeout = %s, want unlimited zero", policy.DefaultTimeout) | |
| 96 | + | } | |
| 97 | + | } | |
| 98 | + | ||
| 75 | 99 | func TestSpawnRunsAndCompletes(t *testing.T) { | |
| 76 | 100 | ran := make(chan string, 1) | |
| 77 | 101 | f := newTestSupervisor(t, func(a *Agent) Runner { | |
M packages/agent/subagents/trace.go +23 -0
| @@ -92,6 +92,7 @@ type TraceWriter struct { | |||
| 92 | 92 | sequence uint64 | |
| 93 | 93 | dropped uint64 | |
| 94 | 94 | events []TraceEvent | |
| 95 | + | projection *traceProjection | |
| 95 | 96 | ||
| 96 | 97 | errorMu sync.Mutex | |
| 97 | 98 | err error | |
| @@ -252,6 +253,13 @@ func (w *TraceWriter) Record(event TraceEvent) { | |||
| 252 | 253 | } | |
| 253 | 254 | w.sequence++ | |
| 254 | 255 | record.event.Seq = w.sequence | |
| 256 | + | if w.projection == nil { | |
| 257 | + | w.projection = newTraceProjection() | |
| 258 | + | } | |
| 259 | + | // Derive live state at ingestion rather than after asynchronous disk I/O. | |
| 260 | + | // This preserves operation boundaries even when the bounded write queue | |
| 261 | + | // drops an old raw event under sustained streaming pressure. | |
| 262 | + | w.projection.apply(record.event) | |
| 255 | 263 | if len(w.pending) >= traceQueueLimit { | |
| 256 | 264 | for index, pending := range w.pending { | |
| 257 | 265 | if pending.barrier == nil { | |
| @@ -375,6 +383,21 @@ func (w *TraceWriter) appendEvent(event TraceEvent) { | |||
| 375 | 383 | } | |
| 376 | 384 | } | |
| 377 | 385 | ||
| 386 | + | // Views returns the incrementally derived trace projection. Unlike replaying | |
| 387 | + | // Events, it retains open operation boundaries after old raw events are evicted | |
| 388 | + | // from the bounded in-memory window. | |
| 389 | + | func (w *TraceWriter) Views() map[string]AgentTraceView { | |
| 390 | + | if w == nil { | |
| 391 | + | return map[string]AgentTraceView{} | |
| 392 | + | } | |
| 393 | + | w.mu.Lock() | |
| 394 | + | defer w.mu.Unlock() | |
| 395 | + | if w.projection == nil { | |
| 396 | + | return map[string]AgentTraceView{} | |
| 397 | + | } | |
| 398 | + | return w.projection.snapshot() | |
| 399 | + | } | |
| 400 | + | ||
| 378 | 401 | // Events returns a stable copy of events already written by this writer. It | |
| 379 | 402 | // never waits for queued filesystem work, keeping status and UI reads cheap. | |
| 380 | 403 | func (w *TraceWriter) Events() []TraceEvent { | |
M packages/agent/subagents/trace_test.go +88 -0
| @@ -9,6 +9,7 @@ import ( | |||
| 9 | 9 | "strings" | |
| 10 | 10 | "sync" | |
| 11 | 11 | "testing" | |
| 12 | + | "time" | |
| 12 | 13 | ) | |
| 13 | 14 | ||
| 14 | 15 | func TestTraceContextZeroValueIsNoOp(t *testing.T) { | |
| @@ -100,6 +101,93 @@ func TestMemoryTraceWriterRetainsOrderedEventsWithoutBundle(t *testing.T) { | |||
| 100 | 101 | } | |
| 101 | 102 | } | |
| 102 | 103 | ||
| 104 | + | func TestSupervisorTraceViewsRetainOpenTurnAcrossEventEviction(t *testing.T) { | |
| 105 | + | writer := NewMemoryTraceWriter() | |
| 106 | + | t.Cleanup(func() { _ = writer.Close() }) | |
| 107 | + | supervisor := &Supervisor{trace: writer} | |
| 108 | + | started := time.Date(2026, time.August, 13, 1, 52, 0, 0, time.UTC) | |
| 109 | + | ||
| 110 | + | writer.Record(TraceEvent{ | |
| 111 | + | Timestamp: started, | |
| 112 | + | Type: "turn.started", | |
| 113 | + | AgentID: "agent-1", | |
| 114 | + | TurnID: "turn-1", | |
| 115 | + | }) | |
| 116 | + | for index := 0; index < memoryTraceEventLimit; index++ { | |
| 117 | + | writer.Record(TraceEvent{ | |
| 118 | + | Timestamp: started.Add(time.Duration(index+1) * time.Millisecond), | |
| 119 | + | Type: "assistant.stream.observed", | |
| 120 | + | AgentID: "agent-1", | |
| 121 | + | TurnID: "turn-1", | |
| 122 | + | }) | |
| 123 | + | if (index+1)%traceQueueInitialCapacity == 0 { | |
| 124 | + | if err := writer.Flush(); err != nil { | |
| 125 | + | t.Fatal(err) | |
| 126 | + | } | |
| 127 | + | } | |
| 128 | + | } | |
| 129 | + | requestStarted := started.Add(time.Minute) | |
| 130 | + | writer.Record(TraceEvent{ | |
| 131 | + | Timestamp: requestStarted, | |
| 132 | + | Type: "provider.request.started", | |
| 133 | + | AgentID: "agent-1", | |
| 134 | + | TurnID: "turn-1", | |
| 135 | + | }) | |
| 136 | + | writer.Record(TraceEvent{ | |
| 137 | + | Timestamp: requestStarted.Add(time.Second), | |
| 138 | + | Type: "provider.request.finished", | |
| 139 | + | AgentID: "agent-1", | |
| 140 | + | TurnID: "turn-1", | |
| 141 | + | }) | |
| 142 | + | if err := writer.Flush(); err != nil { | |
| 143 | + | t.Fatal(err) | |
| 144 | + | } | |
| 145 | + | ||
| 146 | + | view := supervisor.TraceViews()["agent-1"] | |
| 147 | + | if primaryOperation(t, view) == nil || primaryOperation(t, view).Type != "turn.started" { | |
| 148 | + | t.Fatalf("primary operation after request = %#v, want open delegated turn", primaryOperation(t, view)) | |
| 149 | + | } | |
| 150 | + | if !primaryOperation(t, view).StartedAt.Equal(started) { | |
| 151 | + | t.Fatalf("delegated turn started at %v, want %v", primaryOperation(t, view).StartedAt, started) | |
| 152 | + | } | |
| 153 | + | ||
| 154 | + | nextRequestStarted := requestStarted.Add(2 * time.Second) | |
| 155 | + | writer.Record(TraceEvent{ | |
| 156 | + | Timestamp: nextRequestStarted, | |
| 157 | + | Type: "provider.request.started", | |
| 158 | + | AgentID: "agent-1", | |
| 159 | + | TurnID: "turn-1", | |
| 160 | + | }) | |
| 161 | + | if err := writer.Flush(); err != nil { | |
| 162 | + | t.Fatal(err) | |
| 163 | + | } | |
| 164 | + | view = supervisor.TraceViews()["agent-1"] | |
| 165 | + | if primaryOperation(t, view) == nil || primaryOperation(t, view).Type != "provider.request.started" { | |
| 166 | + | t.Fatalf("primary operation during next request = %#v, want provider request", primaryOperation(t, view)) | |
| 167 | + | } | |
| 168 | + | if got := view.TurnStartedAt(*primaryOperation(t, view)); !got.Equal(started) { | |
| 169 | + | t.Fatalf("elapsed timer anchor = %v, want delegated turn start %v", got, started) | |
| 170 | + | } | |
| 171 | + | } | |
| 172 | + | ||
| 173 | + | func TestTraceViewsObserveIngestedBoundariesBeforeAsyncWrite(t *testing.T) { | |
| 174 | + | writer := &TraceWriter{ | |
| 175 | + | pending: make([]traceRecord, 0, traceQueueInitialCapacity), | |
| 176 | + | } | |
| 177 | + | writer.cond = sync.NewCond(&writer.mu) | |
| 178 | + | writer.Record(TraceEvent{ | |
| 179 | + | Timestamp: time.Date(2026, time.August, 13, 2, 0, 0, 0, time.UTC), | |
| 180 | + | Type: "turn.started", | |
| 181 | + | AgentID: "agent-1", | |
| 182 | + | TurnID: "turn-1", | |
| 183 | + | }) | |
| 184 | + | ||
| 185 | + | view := writer.Views()["agent-1"] | |
| 186 | + | if primaryOperation(t, view) == nil || primaryOperation(t, view).Type != "turn.started" { | |
| 187 | + | t.Fatalf("primary operation before async write = %#v, want delegated turn", primaryOperation(t, view)) | |
| 188 | + | } | |
| 189 | + | } | |
| 190 | + | ||
| 103 | 191 | func TestTraceWriterNormalRedactsSensitiveValues(t *testing.T) { | |
| 104 | 192 | writer, err := NewTraceWriter(filepath.Join(t.TempDir(), "trace"), TraceModeNormal) | |
| 105 | 193 | if err != nil { | |
M packages/agent/tools/subagent_resume.go +2 -2
| @@ -11,8 +11,8 @@ import ( | |||
| 11 | 11 | ) | |
| 12 | 12 | ||
| 13 | 13 | // SubagentResumeTool gives a sub-agent a follow-up turn while preserving its | |
| 14 | - | // existing session context. Every explicit follow-up starts a fresh max_turns | |
| 15 | - | // budget. For an idle live worker, the follow-up is delivered directly; for an | |
| 14 | + | // existing session context. Every explicit follow-up starts a fresh observable | |
| 15 | + | // run. For an idle live worker, the follow-up is delivered directly; for an | |
| 16 | 16 | // active live worker, it is queued for the next available message turn; for a | |
| 17 | 17 | // terminal worker, the supervisor restarts the retained session with the | |
| 18 | 18 | // follow-up as its initial turn. | |
M packages/agent/tools/subagent_spawn.go +14 -46
| @@ -1,11 +1,12 @@ | |||
| 1 | 1 | package tools | |
| 2 | 2 | ||
| 3 | 3 | import ( | |
| 4 | + | "bytes" | |
| 4 | 5 | "context" | |
| 5 | 6 | "encoding/json" | |
| 6 | 7 | "fmt" | |
| 8 | + | "io" | |
| 7 | 9 | "strings" | |
| 8 | - | "time" | |
| 9 | 10 | ||
| 10 | 11 | "github.com/bnema/zut/packages/agent/subagents" | |
| 11 | 12 | "github.com/bnema/zut/packages/core" | |
| @@ -62,8 +63,6 @@ type subagentSpawnArgs struct { | |||
| 62 | 63 | FastMode *bool `json:"fast_mode,omitempty"` | |
| 63 | 64 | Required bool `json:"required,omitempty"` | |
| 64 | 65 | Isolation string `json:"isolation,omitempty"` | |
| 65 | - | Timeout string `json:"timeout,omitempty"` | |
| 66 | - | MaxTurns *int `json:"max_turns,omitempty"` | |
| 67 | 66 | } | |
| 68 | 67 | ||
| 69 | 68 | const subagentSpawnSchemaTemplate = `{ | |
| @@ -102,16 +101,6 @@ const subagentSpawnSchemaTemplate = `{ | |||
| 102 | 101 | "type": "string", | |
| 103 | 102 | "enum": ["shared", "worktree"], | |
| 104 | 103 | "description": "Workspace mode. Shared preserves existing behavior; worktree captures a patch without merging it." | |
| 105 | - | }, | |
| 106 | - | "timeout": { | |
| 107 | - | "type": "string", | |
| 108 | - | "description": "Optional Go duration such as 20m." | |
| 109 | - | }, | |
| 110 | - | "max_turns": { | |
| 111 | - | "type": "integer", | |
| 112 | - | "minimum": 1, | |
| 113 | - | "maximum": %d, | |
| 114 | - | "description": "Optional maximum prompt-level turns for this worker. Omit to use the supervisor default; the effective policy allows 1 through %d." | |
| 115 | 104 | } | |
| 116 | 105 | }, | |
| 117 | 106 | "required": ["task"] | |
| @@ -122,13 +111,7 @@ func (t *SubagentSpawnTool) Description() string { | |||
| 122 | 111 | return "Delegate work to a sub-agent. Every spawn returns immediately and completion is host-event-driven through [auto-subagents update]. Set required=true when the outcome is mandatory before the parent's terminal response; failures remain recoverable through subagent_resume. Never use bash sleep, watch, tail -f, polling loops, repeated subagent_status, or dashboard/metadata/event-log/file checks solely to wait. Work on unrelated independent tasks or end/yield your turn. Legitimate waits inside user-requested commands, provider flows, extensions, or tests are allowed." | |
| 123 | 112 | } | |
| 124 | 113 | func (t *SubagentSpawnTool) Schema() json.RawMessage { | |
| 125 | - | maxTurns := 3 | |
| 126 | - | if t.Supervisor != nil { | |
| 127 | - | if limit := t.Supervisor.MaxTurns(); limit > 0 { | |
| 128 | - | maxTurns = limit | |
| 129 | - | } | |
| 130 | - | } | |
| 131 | - | return json.RawMessage(fmt.Sprintf(subagentSpawnSchemaTemplate, maxTurns, maxTurns)) | |
| 114 | + | return json.RawMessage(subagentSpawnSchemaTemplate) | |
| 132 | 115 | } | |
| 133 | 116 | ||
| 134 | 117 | func (t *SubagentSpawnTool) Execute(ctx context.Context, raw json.RawMessage, _ func(string)) (core.ToolResult, error) { | |
| @@ -140,9 +123,14 @@ func (t *SubagentSpawnTool) Execute(ctx context.Context, raw json.RawMessage, _ | |||
| 140 | 123 | return protocolToolError(prefix + ": subagent delegation is unavailable in this mode") | |
| 141 | 124 | } | |
| 142 | 125 | var a subagentSpawnArgs | |
| 143 | - | if err := json.Unmarshal(raw, &a); err != nil { | |
| 126 | + | decoder := json.NewDecoder(bytes.NewReader(raw)) | |
| 127 | + | decoder.DisallowUnknownFields() | |
| 128 | + | if err := decoder.Decode(&a); err != nil { | |
| 144 | 129 | return core.ToolResult{}, fmt.Errorf("invalid args: %w", err) | |
| 145 | 130 | } | |
| 131 | + | if err := decoder.Decode(&struct{}{}); err != io.EOF { | |
| 132 | + | return core.ToolResult{}, fmt.Errorf("invalid args: trailing JSON value") | |
| 133 | + | } | |
| 146 | 134 | task := strings.TrimSpace(a.Task) | |
| 147 | 135 | if task == "" { | |
| 148 | 136 | return protocolToolError(prefix + ": task is required") | |
| @@ -155,23 +143,6 @@ func (t *SubagentSpawnTool) Execute(ctx context.Context, raw json.RawMessage, _ | |||
| 155 | 143 | return protocolToolError(prefix + ": isolation must be shared or worktree") | |
| 156 | 144 | } | |
| 157 | 145 | } | |
| 158 | - | var timeout time.Duration | |
| 159 | - | if value := strings.TrimSpace(a.Timeout); value != "" { | |
| 160 | - | parsed, parseErr := time.ParseDuration(value) | |
| 161 | - | if parseErr != nil || parsed <= 0 { | |
| 162 | - | return protocolToolError(prefix + ": timeout must be a positive duration") | |
| 163 | - | } | |
| 164 | - | timeout = parsed | |
| 165 | - | } | |
| 166 | - | if a.MaxTurns != nil { | |
| 167 | - | if *a.MaxTurns < 1 { | |
| 168 | - | return protocolToolError(prefix + ": max_turns must be positive") | |
| 169 | - | } | |
| 170 | - | if limit := t.Supervisor.MaxTurns(); limit > 0 && *a.MaxTurns > limit { | |
| 171 | - | return protocolToolError(fmt.Sprintf("%s: max_turns must be 1 through %d; omit it to use the supervisor default", prefix, limit)) | |
| 172 | - | } | |
| 173 | - | } | |
| 174 | - | ||
| 175 | 146 | agentName := strings.TrimSpace(a.Agent) | |
| 176 | 147 | var profile *subagents.Profile | |
| 177 | 148 | var fastModeOverride *bool | |
| @@ -242,10 +213,6 @@ func (t *SubagentSpawnTool) Execute(ctx context.Context, raw json.RawMessage, _ | |||
| 242 | 213 | } | |
| 243 | 214 | } | |
| 244 | 215 | ||
| 245 | - | maxTurns := 0 | |
| 246 | - | if a.MaxTurns != nil { | |
| 247 | - | maxTurns = *a.MaxTurns | |
| 248 | - | } | |
| 249 | 216 | agent, err := t.Supervisor.SpawnReq(ctx, subagents.SpawnRequest{ | |
| 250 | 217 | Task: task, | |
| 251 | 218 | Model: model, | |
| @@ -254,8 +221,6 @@ func (t *SubagentSpawnTool) Execute(ctx context.Context, raw json.RawMessage, _ | |||
| 254 | 221 | FastMode: fastModeOverride, | |
| 255 | 222 | Subagent: agentName, | |
| 256 | 223 | Required: a.Required, | |
| 257 | - | Timeout: timeout, | |
| 258 | - | MaxTurns: maxTurns, | |
| 259 | 224 | WorkspaceMode: workspaceMode, | |
| 260 | 225 | Tools: profileTools, | |
| 261 | 226 | }) | |
| @@ -294,6 +259,10 @@ func (t *SubagentSpawnTool) Execute(ctx context.Context, raw json.RawMessage, _ | |||
| 294 | 259 | sb.WriteString("\nThe sub-agent is running in the background. Completion is host-event-driven through [auto-subagents update], the only completion signal; the manager remains free for independent work. ") | |
| 295 | 260 | sb.WriteString("Never use bash sleep, watch, tail -f, polling loops, repeated subagent_status, or dashboard/metadata/event-log/file checks solely to wait. ") | |
| 296 | 261 | sb.WriteString("Work on unrelated independent tasks; otherwise end or yield your turn. Legitimate waits inside user-requested commands, provider flows, extensions, or tests are allowed.") | |
| 262 | + | timeoutDetail := "unlimited" | |
| 263 | + | if agent.Timeout > 0 { | |
| 264 | + | timeoutDetail = agent.Timeout.String() | |
| 265 | + | } | |
| 297 | 266 | details := map[string]any{ | |
| 298 | 267 | "agent_id": agent.ID, | |
| 299 | 268 | "task": task, | |
| @@ -304,8 +273,7 @@ func (t *SubagentSpawnTool) Execute(ctx context.Context, raw json.RawMessage, _ | |||
| 304 | 273 | "fast_mode": agent.FastMode, | |
| 305 | 274 | "required": a.Required, | |
| 306 | 275 | "isolation": string(workspaceMode), | |
| 307 | - | "timeout": agent.Timeout.String(), | |
| 308 | - | "max_turns": agent.MaxTurns, | |
| 276 | + | "timeout": timeoutDetail, | |
| 309 | 277 | "state": agent.Status(), | |
| 310 | 278 | "process_state": string(agent.ProcessState()), | |
| 311 | 279 | "turn_state": string(agent.TurnState()), | |
M packages/agent/tools/subagent_spawn_test.go +34 -108
| @@ -81,52 +81,18 @@ func TestSubagentSpawnGuidanceUsesHostCompletionUpdatesWithoutPolling(t *testing | |||
| 81 | 81 | } | |
| 82 | 82 | } | |
| 83 | 83 | ||
| 84 | - | func TestSubagentSpawnSchemaAdvertisesEffectiveMaxTurns(t *testing.T) { | |
| 85 | - | for _, tc := range []struct { | |
| 86 | - | name string | |
| 87 | - | policy int | |
| 88 | - | want int | |
| 89 | - | }{ | |
| 90 | - | {name: "default", want: 3}, | |
| 91 | - | {name: "negative-default", policy: -1, want: 3}, | |
| 92 | - | {name: "configured", policy: 7, want: 7}, | |
| 93 | - | } { | |
| 94 | - | t.Run(tc.name, func(t *testing.T) { | |
| 95 | - | root := t.TempDir() | |
| 96 | - | manager := subagents.New(subagents.Config{ | |
| 97 | - | Root: filepath.Join(root, "subagents"), | |
| 98 | - | RepoRoot: root, | |
| 99 | - | Policy: subagents.SubagentPolicy{MaxTurns: tc.policy}, | |
| 100 | - | NewRunner: func(*subagents.Agent) subagents.Runner { | |
| 101 | - | return noopSupervisorRunner{} | |
| 102 | - | }, | |
| 103 | - | }) | |
| 104 | - | t.Cleanup(manager.StopAll) | |
| 105 | - | tool := &SubagentSpawnTool{Supervisor: manager} | |
| 106 | - | ||
| 107 | - | var schema struct { | |
| 108 | - | Properties struct { | |
| 109 | - | MaxTurns struct { | |
| 110 | - | Minimum int `json:"minimum"` | |
| 111 | - | Maximum int `json:"maximum"` | |
| 112 | - | Description string `json:"description"` | |
| 113 | - | } `json:"max_turns"` | |
| 114 | - | } `json:"properties"` | |
| 115 | - | } | |
| 116 | - | if err := json.Unmarshal(tool.Schema(), &schema); err != nil { | |
| 117 | - | t.Fatal(err) | |
| 118 | - | } | |
| 119 | - | maxTurns := schema.Properties.MaxTurns | |
| 120 | - | if maxTurns.Minimum != 1 { | |
| 121 | - | t.Fatalf("max_turns minimum = %d, want 1", maxTurns.Minimum) | |
| 122 | - | } | |
| 123 | - | if maxTurns.Maximum != tc.want { | |
| 124 | - | t.Fatalf("max_turns maximum = %d, want %d", maxTurns.Maximum, tc.want) | |
| 125 | - | } | |
| 126 | - | if !strings.Contains(strings.ToLower(maxTurns.Description), "omit") { | |
| 127 | - | t.Fatalf("max_turns description = %q, want omit-to-default guidance", maxTurns.Description) | |
| 128 | - | } | |
| 129 | - | }) | |
| 84 | + | func TestSubagentSpawnSchemaKeepsManagerBudgetsOutOfModelControl(t *testing.T) { | |
| 85 | + | tool := &SubagentSpawnTool{Supervisor: newTestSupervisor(t)} | |
| 86 | + | var schema struct { | |
| 87 | + | Properties map[string]json.RawMessage `json:"properties"` | |
| 88 | + | } | |
| 89 | + | if err := json.Unmarshal(tool.Schema(), &schema); err != nil { | |
| 90 | + | t.Fatal(err) | |
| 91 | + | } | |
| 92 | + | for _, property := range []string{"timeout", "max_turns", "max_steps"} { | |
| 93 | + | if _, found := schema.Properties[property]; found { | |
| 94 | + | t.Fatalf("spawn schema exposes manager-owned property %q", property) | |
| 95 | + | } | |
| 130 | 96 | } | |
| 131 | 97 | } | |
| 132 | 98 | ||
| @@ -159,6 +125,9 @@ func TestSubagentSpawnInheritsHostModelAndProviderWhenOmitted(t *testing.T) { | |||
| 159 | 125 | if got := details["reasoning"]; got != "medium" { | |
| 160 | 126 | t.Fatalf("reasoning detail = %v, want medium", got) | |
| 161 | 127 | } | |
| 128 | + | if got := details["timeout"]; got != "unlimited" { | |
| 129 | + | t.Fatalf("timeout detail = %v, want unlimited", got) | |
| 130 | + | } | |
| 162 | 131 | text := textResult(res.Content) | |
| 163 | 132 | if !strings.Contains(text, "model: gpt-5") || !strings.Contains(text, "provider: openai-codex") || !strings.Contains(text, "reasoning: medium") { | |
| 164 | 133 | t.Fatalf("result text missing inherited model/provider:\n%s", text) | |
| @@ -171,19 +140,15 @@ func TestSubagentSpawnInheritsHostModelAndProviderWhenOmitted(t *testing.T) { | |||
| 171 | 140 | if agents[0].Model != "gpt-5" || agents[0].Provider != "openai-codex" || agents[0].Reasoning != "medium" { | |
| 172 | 141 | t.Fatalf("agent model/provider/reasoning = %q/%q/%q, want gpt-5/openai-codex/medium", agents[0].Model, agents[0].Provider, agents[0].Reasoning) | |
| 173 | 142 | } | |
| 174 | - | if agents[0].MaxTurns != 3 { | |
| 175 | - | t.Fatalf("omitted max_turns = %d, want default 3", agents[0].MaxTurns) | |
| 176 | - | } | |
| 177 | 143 | } | |
| 178 | 144 | ||
| 179 | - | func TestSubagentSpawnDetailsUseEffectiveTimeoutAndTurns(t *testing.T) { | |
| 145 | + | func TestSubagentSpawnDetailsUseEffectiveTimeout(t *testing.T) { | |
| 180 | 146 | root := t.TempDir() | |
| 181 | 147 | manager := subagents.New(subagents.Config{ | |
| 182 | 148 | Root: filepath.Join(root, "subagents"), | |
| 183 | 149 | RepoRoot: root, | |
| 184 | 150 | Policy: subagents.SubagentPolicy{ | |
| 185 | 151 | DefaultTimeout: 37 * time.Minute, | |
| 186 | - | MaxTurns: 7, | |
| 187 | 152 | }, | |
| 188 | 153 | NewRunner: func(*subagents.Agent) subagents.Runner { | |
| 189 | 154 | return noopSupervisorRunner{} | |
| @@ -209,67 +174,25 @@ func TestSubagentSpawnDetailsUseEffectiveTimeoutAndTurns(t *testing.T) { | |||
| 209 | 174 | if got := details["timeout"]; got != (37 * time.Minute).String() { | |
| 210 | 175 | t.Fatalf("omitted timeout detail = %v, want %s", got, (37 * time.Minute).String()) | |
| 211 | 176 | } | |
| 212 | - | if got := details["max_turns"]; got != 7 { | |
| 213 | - | t.Fatalf("max_turns detail = %v, want 7", got) | |
| 177 | + | if _, found := details["max_turns"]; found { | |
| 178 | + | t.Fatalf("details retain removed max_turns: %#v", details) | |
| 214 | 179 | } | |
| 215 | - | } | |
| 216 | - | ||
| 217 | - | func TestSubagentSpawnRejectsExplicitZeroMaxTurns(t *testing.T) { | |
| 218 | - | tool := &SubagentSpawnTool{ | |
| 219 | - | Supervisor: newTestSupervisor(t), | |
| 220 | - | Enabled: func() bool { return true }, | |
| 221 | - | } | |
| 222 | - | ||
| 223 | - | res, err := tool.Execute(context.Background(), json.RawMessage(`{"task":"research docs","max_turns":0}`), nil) | |
| 224 | - | if err != nil { | |
| 225 | - | t.Fatal(err) | |
| 226 | - | } | |
| 227 | - | if !res.IsError || !strings.Contains(textResult(res.Content), "max_turns must be positive") { | |
| 228 | - | t.Fatalf("result = %#v, want max_turns validation error", res) | |
| 229 | - | } | |
| 230 | - | if got := len(tool.Supervisor.List()); got != 0 { | |
| 231 | - | t.Fatalf("spawned agents = %d, want 0", got) | |
| 180 | + | if _, found := details["max_steps"]; found { | |
| 181 | + | t.Fatalf("details retain removed max_steps: %#v", details) | |
| 232 | 182 | } | |
| 233 | 183 | } | |
| 234 | 184 | ||
| 235 | - | func TestSubagentSpawnRejectsMaxTurnsAbovePolicy(t *testing.T) { | |
| 236 | - | for _, tc := range []struct { | |
| 237 | - | name string | |
| 238 | - | policy int | |
| 239 | - | wantLimit string | |
| 240 | - | }{ | |
| 241 | - | {name: "default", wantLimit: "3"}, | |
| 242 | - | {name: "configured", policy: 7, wantLimit: "7"}, | |
| 243 | - | } { | |
| 244 | - | t.Run(tc.name, func(t *testing.T) { | |
| 245 | - | root := t.TempDir() | |
| 246 | - | manager := subagents.New(subagents.Config{ | |
| 247 | - | Root: filepath.Join(root, "subagents"), | |
| 248 | - | RepoRoot: root, | |
| 249 | - | Policy: subagents.SubagentPolicy{MaxTurns: tc.policy}, | |
| 250 | - | NewRunner: func(*subagents.Agent) subagents.Runner { | |
| 251 | - | return noopSupervisorRunner{} | |
| 252 | - | }, | |
| 253 | - | }) | |
| 254 | - | t.Cleanup(manager.StopAll) | |
| 255 | - | tool := &SubagentSpawnTool{ | |
| 256 | - | Supervisor: manager, | |
| 257 | - | Enabled: func() bool { return true }, | |
| 258 | - | } | |
| 259 | - | ||
| 260 | - | res, err := tool.Execute(context.Background(), json.RawMessage(`{"task":"research docs","max_turns":8}`), nil) | |
| 261 | - | if err != nil { | |
| 262 | - | t.Fatal(err) | |
| 263 | - | } | |
| 264 | - | text := textResult(res.Content) | |
| 265 | - | want := "max_turns must be 1 through " + tc.wantLimit | |
| 266 | - | if !res.IsError || !strings.Contains(text, want) || !strings.Contains(text, "omit") { | |
| 267 | - | t.Fatalf("result = %#v, want model-visible policy guidance", res) | |
| 268 | - | } | |
| 269 | - | if got := len(tool.Supervisor.List()); got != 0 { | |
| 270 | - | t.Fatalf("spawned agents = %d, want 0", got) | |
| 271 | - | } | |
| 272 | - | }) | |
| 185 | + | func TestSubagentSpawnRejectsUnknownExecutionLimitFields(t *testing.T) { | |
| 186 | + | manager := newTestSupervisor(t) | |
| 187 | + | tool := &SubagentSpawnTool{Supervisor: manager, Enabled: func() bool { return true }} | |
| 188 | + | for _, field := range []string{"timeout", "max_turns", "max_steps"} { | |
| 189 | + | _, err := tool.Execute(context.Background(), json.RawMessage(fmt.Sprintf(`{"task":"research docs",%q:1}`, field)), nil) | |
| 190 | + | if err == nil || !strings.Contains(err.Error(), "unknown field") { | |
| 191 | + | t.Fatalf("field %q error = %v, want unknown field", field, err) | |
| 192 | + | } | |
| 193 | + | } | |
| 194 | + | if agents := manager.List(); len(agents) != 0 { | |
| 195 | + | t.Fatalf("unknown execution limits spawned agents: %#v", agents) | |
| 273 | 196 | } | |
| 274 | 197 | } | |
| 275 | 198 | ||
| @@ -366,6 +289,9 @@ func TestSubagentSpawnSchemaUsesReasoningAndKeepsProfileThinkingInternal(t *test | |||
| 366 | 289 | if _, ok := schema.Properties["thinking"]; ok { | |
| 367 | 290 | t.Fatal("subagent_spawn schema exposes profile-only thinking") | |
| 368 | 291 | } | |
| 292 | + | if _, ok := schema.Properties["timeout"]; ok { | |
| 293 | + | t.Fatal("subagent_spawn schema exposes manager-owned turn timeout") | |
| 294 | + | } | |
| 369 | 295 | fastMode, ok := schema.Properties["fast_mode"] | |
| 370 | 296 | if !ok { | |
| 371 | 297 | t.Fatal("subagent_spawn schema does not advertise fast_mode") | |
M packages/agent/tools/subagent_status.go +34 -17
| @@ -34,16 +34,17 @@ type subagentStatusResponse struct { | |||
| 34 | 34 | } | |
| 35 | 35 | ||
| 36 | 36 | type subagentStatusEntry struct { | |
| 37 | - | ID string `json:"agent_id"` | |
| 38 | - | Operation *statusOperation `json:"operation,omitempty"` | |
| 39 | - | Terminal string `json:"terminal,omitempty"` | |
| 40 | - | LastEvent *statusLastEvent `json:"last_event,omitempty"` | |
| 41 | - | StartedAt time.Time `json:"started_at"` | |
| 42 | - | LifetimeTurns int `json:"lifetime_turns"` | |
| 43 | - | CurrentRunTurns int `json:"current_run_turns"` | |
| 44 | - | TaskSummary string `json:"task_summary,omitempty"` | |
| 45 | - | Requirement *statusRequirement `json:"requirement,omitempty"` | |
| 46 | - | Result *subagentStatusResult `json:"result,omitempty"` | |
| 37 | + | ID string `json:"agent_id"` | |
| 38 | + | PrimaryOperation *statusOperation `json:"primary_operation,omitempty"` | |
| 39 | + | Terminal string `json:"terminal,omitempty"` | |
| 40 | + | LastObservation *statusLastEvent `json:"last_observation,omitempty"` | |
| 41 | + | LastEvent *statusLastEvent `json:"last_event,omitempty"` | |
| 42 | + | StartedAt time.Time `json:"started_at"` | |
| 43 | + | LifetimeTurns int `json:"lifetime_turns"` | |
| 44 | + | CurrentRunTurns int `json:"current_run_turns"` | |
| 45 | + | TaskSummary string `json:"task_summary,omitempty"` | |
| 46 | + | Requirement *statusRequirement `json:"requirement,omitempty"` | |
| 47 | + | Result *subagentStatusResult `json:"result,omitempty"` | |
| 47 | 48 | } | |
| 48 | 49 | ||
| 49 | 50 | type statusOperation struct { | |
| @@ -64,9 +65,11 @@ type statusRequirement struct { | |||
| 64 | 65 | } | |
| 65 | 66 | ||
| 66 | 67 | type subagentStatusResult struct { | |
| 67 | - | State string `json:"state"` | |
| 68 | - | Available bool `json:"available"` | |
| 69 | - | Ref string `json:"ref,omitempty"` | |
| 68 | + | State string `json:"state"` | |
| 69 | + | Available bool `json:"available"` | |
| 70 | + | Ref string `json:"ref,omitempty"` | |
| 71 | + | Delivered bool `json:"delivered"` | |
| 72 | + | DeliveryFailed bool `json:"delivery_failed"` | |
| 70 | 73 | } | |
| 71 | 74 | ||
| 72 | 75 | const subagentStatusSchema = `{ | |
| @@ -157,9 +160,11 @@ func publicSubagentStatus(snapshot subagents.AgentSnapshot, view subagents.Agent | |||
| 157 | 160 | CurrentRunTurns: snapshot.CurrentRunTurns, | |
| 158 | 161 | TaskSummary: summarizeSubagentTask(snapshot.Task), | |
| 159 | 162 | } | |
| 160 | - | if len(view.OpenOperations) != 0 { | |
| 161 | - | operation := view.OpenOperations[0] | |
| 162 | - | entry.Operation = &statusOperation{Type: operation.Type, StartedAt: operation.StartedAt} | |
| 163 | + | if primary, ok := view.Primary(); ok { | |
| 164 | + | entry.PrimaryOperation = &statusOperation{Type: primary.Type, StartedAt: primary.StartedAt} | |
| 165 | + | } | |
| 166 | + | if view.LastObservation != nil { | |
| 167 | + | entry.LastObservation = &statusLastEvent{Type: view.LastObservation.Type, At: view.LastObservation.At} | |
| 163 | 168 | } | |
| 164 | 169 | if view.LastEvent.Type != "" { | |
| 165 | 170 | entry.LastEvent = &statusLastEvent{Type: view.LastEvent.Type, At: view.LastEvent.Timestamp} | |
| @@ -175,9 +180,21 @@ func publicSubagentStatus(snapshot subagents.AgentSnapshot, view subagents.Agent | |||
| 175 | 180 | if snapshot.Result != nil { | |
| 176 | 181 | entry.Result = &subagentStatusResult{ | |
| 177 | 182 | State: publicResultState(snapshot.Result.Status), | |
| 178 | - | Available: true, | |
| 183 | + | Available: snapshot.ResultRef != "", | |
| 179 | 184 | Ref: snapshot.ResultRef, | |
| 185 | + | Delivered: snapshot.Requirement.Notified, | |
| 186 | + | } | |
| 187 | + | } | |
| 188 | + | if view.Result != nil { | |
| 189 | + | if entry.Result == nil { | |
| 190 | + | entry.Result = &subagentStatusResult{State: "unknown"} | |
| 191 | + | } | |
| 192 | + | entry.Result.Available = view.Result.Available | |
| 193 | + | if view.Result.Ref != "" { | |
| 194 | + | entry.Result.Ref = view.Result.Ref | |
| 180 | 195 | } | |
| 196 | + | entry.Result.Delivered = entry.Result.Delivered || view.Result.Delivered | |
| 197 | + | entry.Result.DeliveryFailed = entry.Result.DeliveryFailed || view.Result.Failed | |
| 181 | 198 | } | |
| 182 | 199 | return entry | |
| 183 | 200 | } | |
M packages/agent/tools/subagent_status_test.go +73 -0
| @@ -61,6 +61,79 @@ func TestSubagentStatusIncludesTurnCounters(t *testing.T) { | |||
| 61 | 61 | } | |
| 62 | 62 | } | |
| 63 | 63 | ||
| 64 | + | func TestPublicSubagentStatusUsesOrderedPrimaryOperation(t *testing.T) { | |
| 65 | + | at := time.Date(2026, 8, 12, 7, 0, 0, 0, time.UTC) | |
| 66 | + | entry := publicSubagentStatus(subagents.AgentSnapshot{ID: "agent-1"}, subagents.AgentTraceView{ | |
| 67 | + | OpenOperations: []subagents.Operation{ | |
| 68 | + | {Type: "tool.started", StartedAt: at.Add(time.Second)}, | |
| 69 | + | {Type: "turn.started", StartedAt: at}, | |
| 70 | + | }, | |
| 71 | + | }) | |
| 72 | + | if entry.PrimaryOperation == nil || entry.PrimaryOperation.Type != "tool.started" { | |
| 73 | + | t.Fatalf("primary operation = %#v", entry.PrimaryOperation) | |
| 74 | + | } | |
| 75 | + | encoded, err := json.Marshal(entry) | |
| 76 | + | if err != nil { | |
| 77 | + | t.Fatal(err) | |
| 78 | + | } | |
| 79 | + | var fields map[string]any | |
| 80 | + | if err := json.Unmarshal(encoded, &fields); err != nil { | |
| 81 | + | t.Fatal(err) | |
| 82 | + | } | |
| 83 | + | if _, found := fields["operation"]; found { | |
| 84 | + | t.Fatalf("obsolete operation field in %#v", fields) | |
| 85 | + | } | |
| 86 | + | if _, found := fields["primary_operation"]; !found { | |
| 87 | + | t.Fatalf("primary operation missing from %#v", fields) | |
| 88 | + | } | |
| 89 | + | } | |
| 90 | + | ||
| 91 | + | func TestPublicSubagentStatusDoesNotInferAvailabilityWithoutResultReference(t *testing.T) { | |
| 92 | + | entry := publicSubagentStatus(subagents.AgentSnapshot{ | |
| 93 | + | ID: "agent-1", | |
| 94 | + | Result: &subagents.TurnResult{Status: subagents.ResultFailed}, | |
| 95 | + | }, subagents.AgentTraceView{}) | |
| 96 | + | if entry.Result == nil || entry.Result.Available { | |
| 97 | + | t.Fatalf("result availability = %#v, want unavailable without result ref", entry.Result) | |
| 98 | + | } | |
| 99 | + | } | |
| 100 | + | ||
| 101 | + | func TestPublicSubagentStatusIncludesResultDeliveryFacts(t *testing.T) { | |
| 102 | + | entry := publicSubagentStatus(subagents.AgentSnapshot{ID: "agent-1"}, subagents.AgentTraceView{ | |
| 103 | + | Result: &subagents.ResultFact{Available: true, Ref: "subagent://agent-1/result"}, | |
| 104 | + | }) | |
| 105 | + | if entry.Result == nil || entry.Result.State != "unknown" || entry.Result.Delivered || entry.Result.DeliveryFailed { | |
| 106 | + | t.Fatalf("result = %#v", entry.Result) | |
| 107 | + | } | |
| 108 | + | encoded, err := json.Marshal(entry) | |
| 109 | + | if err != nil { | |
| 110 | + | t.Fatal(err) | |
| 111 | + | } | |
| 112 | + | var fields map[string]json.RawMessage | |
| 113 | + | if err := json.Unmarshal(encoded, &fields); err != nil { | |
| 114 | + | t.Fatal(err) | |
| 115 | + | } | |
| 116 | + | var result map[string]any | |
| 117 | + | if err := json.Unmarshal(fields["result"], &result); err != nil { | |
| 118 | + | t.Fatal(err) | |
| 119 | + | } | |
| 120 | + | if result["delivered"] != false || result["delivery_failed"] != false { | |
| 121 | + | t.Fatalf("result JSON = %#v", result) | |
| 122 | + | } | |
| 123 | + | } | |
| 124 | + | ||
| 125 | + | func TestPublicSubagentStatusKeepsDurableDeliveryAfterTraceRestart(t *testing.T) { | |
| 126 | + | entry := publicSubagentStatus(subagents.AgentSnapshot{ | |
| 127 | + | ID: "agent-1", | |
| 128 | + | Result: &subagents.TurnResult{Status: subagents.ResultSucceeded}, | |
| 129 | + | ResultRef: "subagent://agent-1/result", | |
| 130 | + | Requirement: subagents.RequirementSnapshot{Required: true, Notified: true}, | |
| 131 | + | }, subagents.AgentTraceView{}) | |
| 132 | + | if entry.Result == nil || !entry.Result.Available || !entry.Result.Delivered { | |
| 133 | + | t.Fatalf("result = %#v, want durable delivered fact without transient trace", entry.Result) | |
| 134 | + | } | |
| 135 | + | } | |
| 136 | + | ||
| 64 | 137 | func TestSubagentStatusSchemaHasOptionalAgentID(t *testing.T) { | |
| 65 | 138 | var schema struct { | |
| 66 | 139 | Type string `json:"type"` | |
M packages/core/agent.go +107 -15
| @@ -13,11 +13,29 @@ import ( | |||
| 13 | 13 | "github.com/bnema/zut/packages/provider" | |
| 14 | 14 | ) | |
| 15 | 15 | ||
| 16 | + | var ( | |
| 17 | + | ErrStreamIdleTimeout = errors.New("provider stream idle timeout") | |
| 18 | + | ErrMaxSteps = errors.New("agent model step limit exceeded") | |
| 19 | + | ) | |
| 20 | + | ||
| 21 | + | const ( | |
| 22 | + | DefaultStreamIdleTimeout = 5 * time.Minute | |
| 23 | + | DefaultStreamMaxRetries = 5 | |
| 24 | + | ) | |
| 25 | + | ||
| 16 | 26 | type queuedMessage struct { | |
| 17 | 27 | text string | |
| 18 | 28 | accepted time.Time | |
| 19 | 29 | } | |
| 20 | 30 | ||
| 31 | + | // requestOpenError marks a failure returned before a provider stream exists. | |
| 32 | + | // Provider clients own retries for this phase; the agent retry budget is only | |
| 33 | + | // for reconnecting a stream that opened and then failed. | |
| 34 | + | type requestOpenError struct{ err error } | |
| 35 | + | ||
| 36 | + | func (e *requestOpenError) Error() string { return e.err.Error() } | |
| 37 | + | func (e *requestOpenError) Unwrap() error { return e.err } | |
| 38 | + | ||
| 21 | 39 | // Agent is a stateful conversation bound to a provider client, a model, | |
| 22 | 40 | // and a set of tools. | |
| 23 | 41 | type Agent struct { | |
| @@ -89,12 +107,15 @@ type Agent struct { | |||
| 89 | 107 | // model can still see what it said in subsequent turns). | |
| 90 | 108 | BeforeAssistantMessage func(text string) (allowed bool, reason, replacement string) | |
| 91 | 109 | ||
| 92 | - | // MaxRetries controls agent-level retries for transient provider | |
| 93 | - | // failures that arrive after the HTTP stream opens (for example | |
| 94 | - | // Anthropic overloaded_error). Zero disables this retry layer. | |
| 110 | + | // MaxRetries controls reconnection attempts for transient failures after the | |
| 111 | + | // provider stream opens. Zero disables this retry layer. | |
| 95 | 112 | // RetryBaseDelay is doubled for each attempt; zero uses 2s. | |
| 96 | 113 | MaxRetries int | |
| 97 | 114 | RetryBaseDelay time.Duration | |
| 115 | + | // StreamIdleTimeout cancels an individual provider stream when it emits no | |
| 116 | + | // events for this duration. Zero leaves the stream unbounded. | |
| 117 | + | StreamIdleTimeout time.Duration | |
| 118 | + | streamIdleTimer func(time.Duration) (<-chan time.Time, func(), func()) | |
| 98 | 119 | ||
| 99 | 120 | // OnEvent, if set, mirrors every AgentEvent the loop emits to | |
| 100 | 121 | // this callback in addition to the per-Prompt sink. Used by the | |
| @@ -149,14 +170,15 @@ type Agent struct { | |||
| 149 | 170 | // NewAgent returns an Agent with sensible defaults. | |
| 150 | 171 | func NewAgent(client provider.Client, model, system string, tools Registry) *Agent { | |
| 151 | 172 | return &Agent{ | |
| 152 | - | Client: client, | |
| 153 | - | Model: model, | |
| 154 | - | System: system, | |
| 155 | - | Tools: tools, | |
| 156 | - | MaxSteps: 0, // 0 = unlimited | |
| 157 | - | MaxRetries: 3, | |
| 158 | - | RetryBaseDelay: 2 * time.Second, | |
| 159 | - | timeContext: newAgentTimeContext(time.Now()), | |
| 173 | + | Client: client, | |
| 174 | + | Model: model, | |
| 175 | + | System: system, | |
| 176 | + | Tools: tools, | |
| 177 | + | MaxSteps: 0, // 0 = unlimited | |
| 178 | + | MaxRetries: DefaultStreamMaxRetries, | |
| 179 | + | RetryBaseDelay: 2 * time.Second, | |
| 180 | + | StreamIdleTimeout: DefaultStreamIdleTimeout, | |
| 181 | + | timeContext: newAgentTimeContext(time.Now()), | |
| 160 | 182 | } | |
| 161 | 183 | } | |
| 162 | 184 | ||
| @@ -645,7 +667,7 @@ func (a *Agent) runLoop(ctx context.Context, sink func(AgentEvent)) error { | |||
| 645 | 667 | } | |
| 646 | 668 | if a.MaxSteps > 0 { | |
| 647 | 669 | sink(EvDone{}) | |
| 648 | - | return fmt.Errorf("max steps (%d) exceeded", a.MaxSteps) | |
| 670 | + | return fmt.Errorf("%w: %d", ErrMaxSteps, a.MaxSteps) | |
| 649 | 671 | } | |
| 650 | 672 | return nil | |
| 651 | 673 | } | |
| @@ -654,6 +676,10 @@ func (a *Agent) canRetryError(err error, attempt int) bool { | |||
| 654 | 676 | if err == nil || a.MaxRetries <= 0 || attempt >= a.MaxRetries { | |
| 655 | 677 | return false | |
| 656 | 678 | } | |
| 679 | + | var openErr *requestOpenError | |
| 680 | + | if errors.As(err, &openErr) { | |
| 681 | + | return false | |
| 682 | + | } | |
| 657 | 683 | if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { | |
| 658 | 684 | return false | |
| 659 | 685 | } | |
| @@ -866,6 +892,21 @@ func retryReasonFromProvider(reason provider.RequestFailureReason) RetryReason { | |||
| 866 | 892 | ||
| 867 | 893 | // oneTurn calls the LLM once, forwards events, returns the stop reason | |
| 868 | 894 | // and the assembled assistant message (already appended to the transcript). | |
| 895 | + | func newStreamIdleTimer(timeout time.Duration) (<-chan time.Time, func(), func()) { | |
| 896 | + | timer := time.NewTimer(timeout) | |
| 897 | + | stop := func() { timer.Stop() } | |
| 898 | + | reset := func() { | |
| 899 | + | if !timer.Stop() { | |
| 900 | + | select { | |
| 901 | + | case <-timer.C: | |
| 902 | + | default: | |
| 903 | + | } | |
| 904 | + | } | |
| 905 | + | timer.Reset(timeout) | |
| 906 | + | } | |
| 907 | + | return timer.C, stop, reset | |
| 908 | + | } | |
| 909 | + | ||
| 869 | 910 | func (a *Agent) oneTurn(ctx context.Context, sink func(AgentEvent), turnContext string, attempt, maxAttempts int) (provider.StopReason, provider.Message, error) { | |
| 870 | 911 | fastMode := a.FastModeEnabled() | |
| 871 | 912 | if err := provider.ValidateFastMode(a.Client.Name(), fastMode); err != nil { | |
| @@ -916,10 +957,16 @@ func (a *Agent) oneTurn(ctx context.Context, sink func(AgentEvent), turnContext | |||
| 916 | 957 | Attempt: attempt + 1, | |
| 917 | 958 | MaxAttempts: maxAttempts, | |
| 918 | 959 | }) | |
| 919 | - | stream, err := a.Client.Stream(ctx, req) | |
| 960 | + | streamCtx, cancelStream := context.WithCancel(ctx) | |
| 961 | + | defer cancelStream() | |
| 962 | + | stream, err := a.Client.Stream(streamCtx, req) | |
| 920 | 963 | if err != nil { | |
| 921 | - | return provider.StopError, provider.Message{}, err | |
| 964 | + | return provider.StopError, provider.Message{}, &requestOpenError{err: err} | |
| 922 | 965 | } | |
| 966 | + | // A provider may already have queued events when cancellation wins. Keep a | |
| 967 | + | // consumer attached after this turn returns so bounded producer channels | |
| 968 | + | // cannot strand the provider goroutine and its response body. | |
| 969 | + | defer func() { go drainProviderStream(stream) }() | |
| 923 | 970 | ||
| 924 | 971 | sink(EvAssistantStart{}) | |
| 925 | 972 | ||
| @@ -929,7 +976,21 @@ func (a *Agent) oneTurn(ctx context.Context, sink func(AgentEvent), turnContext | |||
| 929 | 976 | finalMsg provider.Message | |
| 930 | 977 | ) | |
| 931 | 978 | ||
| 932 | - | for ev := range stream { | |
| 979 | + | var idle <-chan time.Time | |
| 980 | + | var resetIdle func() | |
| 981 | + | if a.StreamIdleTimeout > 0 { | |
| 982 | + | newTimer := a.streamIdleTimer | |
| 983 | + | if newTimer == nil { | |
| 984 | + | newTimer = newStreamIdleTimer | |
| 985 | + | } | |
| 986 | + | var stopIdle func() | |
| 987 | + | idle, stopIdle, resetIdle = newTimer(a.StreamIdleTimeout) | |
| 988 | + | defer stopIdle() | |
| 989 | + | } | |
| 990 | + | handleStreamEvent := func(ev provider.Event) (done bool) { | |
| 991 | + | if resetIdle != nil { | |
| 992 | + | resetIdle() | |
| 993 | + | } | |
| 933 | 994 | switch e := ev.(type) { | |
| 934 | 995 | case provider.EventStart: | |
| 935 | 996 | // nothing | |
| @@ -951,8 +1012,34 @@ func (a *Agent) oneTurn(ctx context.Context, sink func(AgentEvent), turnContext | |||
| 951 | 1012 | stop = e.Stop | |
| 952 | 1013 | finalErr = e.Err | |
| 953 | 1014 | finalMsg = e.Message | |
| 1015 | + | return true | |
| 1016 | + | } | |
| 1017 | + | return false | |
| 1018 | + | } | |
| 1019 | + | for { | |
| 1020 | + | // Consume no more than one buffered event before honoring cancellation. | |
| 1021 | + | select { | |
| 1022 | + | case ev, ok := <-stream: | |
| 1023 | + | if !ok || handleStreamEvent(ev) { | |
| 1024 | + | goto streamDone | |
| 1025 | + | } | |
| 1026 | + | default: | |
| 1027 | + | } | |
| 1028 | + | if err := streamCtx.Err(); err != nil { | |
| 1029 | + | return provider.StopError, finalMsg, err | |
| 1030 | + | } | |
| 1031 | + | select { | |
| 1032 | + | case <-streamCtx.Done(): | |
| 1033 | + | return provider.StopError, finalMsg, streamCtx.Err() | |
| 1034 | + | case <-idle: | |
| 1035 | + | return provider.StopError, finalMsg, ErrStreamIdleTimeout | |
| 1036 | + | case ev, ok := <-stream: | |
| 1037 | + | if !ok || handleStreamEvent(ev) { | |
| 1038 | + | goto streamDone | |
| 1039 | + | } | |
| 954 | 1040 | } | |
| 955 | 1041 | } | |
| 1042 | + | streamDone: | |
| 956 | 1043 | ||
| 957 | 1044 | // Append assistant message to transcript. Aborted turns (Esc / Ctrl+C) | |
| 958 | 1045 | // produce partial content. When the partial message is text only we | |
| @@ -1014,6 +1101,11 @@ func (a *Agent) oneTurn(ctx context.Context, sink func(AgentEvent), turnContext | |||
| 1014 | 1101 | return stop, finalMsg, finalErr | |
| 1015 | 1102 | } | |
| 1016 | 1103 | ||
| 1104 | + | func drainProviderStream(stream <-chan provider.Event) { | |
| 1105 | + | for range stream { | |
| 1106 | + | } | |
| 1107 | + | } | |
| 1108 | + | ||
| 1017 | 1109 | // executeTools runs every tool call in the assistant message and returns | |
| 1018 | 1110 | // a single tool-role message carrying all results. | |
| 1019 | 1111 | func (a *Agent) executeTools(ctx context.Context, msg provider.Message, sink func(AgentEvent)) (provider.Message, bool) { | |
M packages/core/agent_retry_test.go +146 -5
| @@ -48,6 +48,133 @@ func (c *retryFakeClient) Stream(ctx context.Context, req provider.Request) (<-c | |||
| 48 | 48 | return out, nil | |
| 49 | 49 | } | |
| 50 | 50 | ||
| 51 | + | type silentStreamClient struct{} | |
| 52 | + | ||
| 53 | + | func (silentStreamClient) Name() string { return "silent-stream" } | |
| 54 | + | ||
| 55 | + | func (silentStreamClient) Stream(ctx context.Context, _ provider.Request) (<-chan provider.Event, error) { | |
| 56 | + | out := make(chan provider.Event) | |
| 57 | + | go func() { | |
| 58 | + | defer close(out) | |
| 59 | + | <-ctx.Done() | |
| 60 | + | }() | |
| 61 | + | return out, nil | |
| 62 | + | } | |
| 63 | + | ||
| 64 | + | type activeStreamClient struct{} | |
| 65 | + | ||
| 66 | + | func (activeStreamClient) Name() string { return "active-stream" } | |
| 67 | + | ||
| 68 | + | func (activeStreamClient) Stream(context.Context, provider.Request) (<-chan provider.Event, error) { | |
| 69 | + | out := make(chan provider.Event, 2) | |
| 70 | + | out <- provider.EventStart{} | |
| 71 | + | out <- provider.EventDone{Stop: provider.StopEnd} | |
| 72 | + | close(out) | |
| 73 | + | return out, nil | |
| 74 | + | } | |
| 75 | + | ||
| 76 | + | type requestOpenFailureClient struct{ calls int32 } | |
| 77 | + | ||
| 78 | + | func (c *requestOpenFailureClient) Name() string { return "request-open-failure" } | |
| 79 | + | ||
| 80 | + | func (c *requestOpenFailureClient) Stream(context.Context, provider.Request) (<-chan provider.Event, error) { | |
| 81 | + | atomic.AddInt32(&c.calls, 1) | |
| 82 | + | return nil, errors.New("provider returned error: 503 service unavailable") | |
| 83 | + | } | |
| 84 | + | ||
| 85 | + | type readableAfterCancellationClient struct { | |
| 86 | + | cancel context.CancelFunc | |
| 87 | + | drained chan struct{} | |
| 88 | + | } | |
| 89 | + | ||
| 90 | + | func (c readableAfterCancellationClient) Name() string { return "readable-after-cancellation" } | |
| 91 | + | ||
| 92 | + | func (c readableAfterCancellationClient) Stream(_ context.Context, _ provider.Request) (<-chan provider.Event, error) { | |
| 93 | + | out := make(chan provider.Event, 1) | |
| 94 | + | out <- provider.EventStart{} | |
| 95 | + | c.cancel() | |
| 96 | + | go func() { | |
| 97 | + | out <- provider.EventTextDelta{Delta: "discard after cancellation"} | |
| 98 | + | close(out) | |
| 99 | + | close(c.drained) | |
| 100 | + | }() | |
| 101 | + | return out, nil | |
| 102 | + | } | |
| 103 | + | ||
| 104 | + | func TestNewAgentUsesCodexStreamGuardDefaults(t *testing.T) { | |
| 105 | + | a := NewAgent(&retryFakeClient{}, "fake-model", "system", Registry{}) | |
| 106 | + | if a.MaxSteps != 0 { | |
| 107 | + | t.Fatalf("max steps = %d, want unlimited", a.MaxSteps) | |
| 108 | + | } | |
| 109 | + | if a.MaxRetries != 5 { | |
| 110 | + | t.Fatalf("stream retries = %d, want 5", a.MaxRetries) | |
| 111 | + | } | |
| 112 | + | if a.StreamIdleTimeout != 5*time.Minute { | |
| 113 | + | t.Fatalf("stream idle timeout = %s, want 5m", a.StreamIdleTimeout) | |
| 114 | + | } | |
| 115 | + | } | |
| 116 | + | ||
| 117 | + | func TestAgentStopsReadingStreamAfterCancellation(t *testing.T) { | |
| 118 | + | ctx, cancel := context.WithCancel(context.Background()) | |
| 119 | + | drained := make(chan struct{}) | |
| 120 | + | a := NewAgent(readableAfterCancellationClient{cancel: cancel, drained: drained}, "fake-model", "system", Registry{}) | |
| 121 | + | a.MaxRetries = 0 | |
| 122 | + | if err := a.Prompt(ctx, "hello", nil, nil); !errors.Is(err, context.Canceled) { | |
| 123 | + | t.Fatalf("Prompt error = %v, want context.Canceled", err) | |
| 124 | + | } | |
| 125 | + | select { | |
| 126 | + | case <-drained: | |
| 127 | + | case <-time.After(time.Second): | |
| 128 | + | t.Fatal("provider producer remained blocked after turn cancellation") | |
| 129 | + | } | |
| 130 | + | } | |
| 131 | + | ||
| 132 | + | func TestAgentStopsSilentStreamAtIdleDeadline(t *testing.T) { | |
| 133 | + | a := NewAgent(silentStreamClient{}, "fake-model", "system", Registry{}) | |
| 134 | + | a.MaxRetries = 0 | |
| 135 | + | a.StreamIdleTimeout = time.Minute | |
| 136 | + | idle := make(chan time.Time, 1) | |
| 137 | + | a.streamIdleTimer = func(time.Duration) (<-chan time.Time, func(), func()) { | |
| 138 | + | idle <- time.Now() | |
| 139 | + | return idle, func() {}, func() {} | |
| 140 | + | } | |
| 141 | + | if err := a.Prompt(context.Background(), "hello", nil, nil); !errors.Is(err, ErrStreamIdleTimeout) { | |
| 142 | + | t.Fatalf("Prompt error = %v, want ErrStreamIdleTimeout", err) | |
| 143 | + | } | |
| 144 | + | } | |
| 145 | + | ||
| 146 | + | func TestAgentResetsIdleDeadlineForEveryStreamEvent(t *testing.T) { | |
| 147 | + | a := NewAgent(activeStreamClient{}, "fake-model", "system", Registry{}) | |
| 148 | + | a.MaxRetries = 0 | |
| 149 | + | a.StreamIdleTimeout = time.Minute | |
| 150 | + | var resets atomic.Int32 | |
| 151 | + | a.streamIdleTimer = func(time.Duration) (<-chan time.Time, func(), func()) { | |
| 152 | + | return make(chan time.Time), func() {}, func() { resets.Add(1) } | |
| 153 | + | } | |
| 154 | + | if err := a.Prompt(context.Background(), "hello", nil, nil); err != nil { | |
| 155 | + | t.Fatal(err) | |
| 156 | + | } | |
| 157 | + | if got := resets.Load(); got != 2 { | |
| 158 | + | t.Fatalf("idle timer resets = %d, want one for each stream event", got) | |
| 159 | + | } | |
| 160 | + | } | |
| 161 | + | ||
| 162 | + | func TestAgentRetriesSilentStreamWithoutEndingTurn(t *testing.T) { | |
| 163 | + | client := &retryFakeClient{firstErr: ErrStreamIdleTimeout} | |
| 164 | + | a := NewAgent(client, "fake-model", "system", Registry{}) | |
| 165 | + | a.MaxRetries = 1 | |
| 166 | + | a.RetryBaseDelay = time.Millisecond | |
| 167 | + | if err := a.Prompt(context.Background(), "hello", nil, nil); err != nil { | |
| 168 | + | t.Fatalf("Prompt returned %v", err) | |
| 169 | + | } | |
| 170 | + | if got := atomic.LoadInt32(&client.calls); got != 2 { | |
| 171 | + | t.Fatalf("Stream calls = %d; want reconnect after idle timeout", got) | |
| 172 | + | } | |
| 173 | + | if got := extractText(a.Messages()[1]); got != "ok" { | |
| 174 | + | t.Fatalf("final assistant text = %q; want ok", got) | |
| 175 | + | } | |
| 176 | + | } | |
| 177 | + | ||
| 51 | 178 | func TestAgentRetriesOverloadedStreamError(t *testing.T) { | |
| 52 | 179 | client := &retryFakeClient{} | |
| 53 | 180 | a := NewAgent(client, "fake-model", "system", Registry{}) | |
| @@ -77,6 +204,20 @@ func TestAgentRetriesOverloadedStreamError(t *testing.T) { | |||
| 77 | 204 | } | |
| 78 | 205 | } | |
| 79 | 206 | ||
| 207 | + | func TestAgentDoesNotReplayProviderRequestRetriesAtStreamLayer(t *testing.T) { | |
| 208 | + | client := &requestOpenFailureClient{} | |
| 209 | + | a := NewAgent(client, "fake-model", "system", Registry{}) | |
| 210 | + | a.RetryBaseDelay = time.Millisecond | |
| 211 | + | ||
| 212 | + | err := a.Prompt(context.Background(), "hello", nil, nil) | |
| 213 | + | if err == nil || !strings.Contains(err.Error(), "503") { | |
| 214 | + | t.Fatalf("Prompt error = %v, want provider failure", err) | |
| 215 | + | } | |
| 216 | + | if got := atomic.LoadInt32(&client.calls); got != 1 { | |
| 217 | + | t.Fatalf("Stream calls = %d, want one provider-owned request sequence", got) | |
| 218 | + | } | |
| 219 | + | } | |
| 220 | + | ||
| 80 | 221 | func TestAgentRetriesUnexpectedEOFStreamError(t *testing.T) { | |
| 81 | 222 | client := &retryFakeClient{firstErr: fmt.Errorf("read SSE: %w", io.ErrUnexpectedEOF)} | |
| 82 | 223 | a := NewAgent(client, "fake-model", "system", Registry{}) | |
| @@ -108,10 +249,10 @@ func TestAgentEmitsRetryLifecycleEvents(t *testing.T) { | |||
| 108 | 249 | } | |
| 109 | 250 | ||
| 110 | 251 | want := []string{ | |
| 111 | - | "request:agent:1/4", | |
| 252 | + | "request:agent:1/6", | |
| 112 | 253 | "request:provider:1/1", | |
| 113 | - | "retry:agent:2/4", | |
| 114 | - | "request:agent:2/4", | |
| 254 | + | "retry:agent:2/6", | |
| 255 | + | "request:agent:2/6", | |
| 115 | 256 | "request:provider:1/1", | |
| 116 | 257 | } | |
| 117 | 258 | if strings.Join(lifecycle, ",") != strings.Join(want, ",") { | |
| @@ -135,11 +276,11 @@ func TestAgentReportsSanitizedRetryLifecycle(t *testing.T) { | |||
| 135 | 276 | want := []RetryLifecycleRecord{ | |
| 136 | 277 | { | |
| 137 | 278 | Event: RetryLifecycleRequestFailed, Scope: RetryScopeAgent, | |
| 138 | - | Attempt: 1, MaxAttempts: 4, Reason: RetryReasonOverload, | |
| 279 | + | Attempt: 1, MaxAttempts: 6, Reason: RetryReasonOverload, | |
| 139 | 280 | }, | |
| 140 | 281 | { | |
| 141 | 282 | Event: RetryLifecycleRetryScheduled, Scope: RetryScopeAgent, | |
| 142 | - | Attempt: 2, MaxAttempts: 4, Reason: RetryReasonOverload, DelayMS: 1, | |
| 283 | + | Attempt: 2, MaxAttempts: 6, Reason: RetryReasonOverload, DelayMS: 1, | |
| 143 | 284 | }, | |
| 144 | 285 | } | |
| 145 | 286 | if !reflect.DeepEqual(records, want) { | |
A packages/core/agent_step_limit_test.go +49 -0
| @@ -0,0 +1,49 @@ | |||
| 1 | + | package core | |
| 2 | + | ||
| 3 | + | import ( | |
| 4 | + | "context" | |
| 5 | + | "encoding/json" | |
| 6 | + | "errors" | |
| 7 | + | "fmt" | |
| 8 | + | "sync/atomic" | |
| 9 | + | "testing" | |
| 10 | + | ||
| 11 | + | "github.com/bnema/zut/packages/provider" | |
| 12 | + | ) | |
| 13 | + | ||
| 14 | + | type loopingToolClient struct { | |
| 15 | + | calls int32 | |
| 16 | + | } | |
| 17 | + | ||
| 18 | + | func (c *loopingToolClient) Name() string { return "looping-tool" } | |
| 19 | + | ||
| 20 | + | func (c *loopingToolClient) Stream(_ context.Context, req provider.Request) (<-chan provider.Event, error) { | |
| 21 | + | call := atomic.AddInt32(&c.calls, 1) | |
| 22 | + | callID := fmt.Sprintf("call-%d", call) | |
| 23 | + | out := make(chan provider.Event, 4) | |
| 24 | + | out <- provider.EventStart{Provider: c.Name(), Model: req.Model} | |
| 25 | + | out <- provider.EventToolStart{ID: callID, Name: "echo"} | |
| 26 | + | out <- provider.EventToolEnd{ID: callID} | |
| 27 | + | out <- provider.EventDone{Stop: provider.StopToolUse, Message: provider.Message{ | |
| 28 | + | Role: provider.RoleAssistant, | |
| 29 | + | Content: []provider.Content{provider.ToolCallBlock{ | |
| 30 | + | ID: callID, Name: "echo", Arguments: json.RawMessage(`{}`), | |
| 31 | + | }}, | |
| 32 | + | }} | |
| 33 | + | close(out) | |
| 34 | + | return out, nil | |
| 35 | + | } | |
| 36 | + | ||
| 37 | + | func TestAgentReturnsTypedStepLimitAfterModelLoopBudget(t *testing.T) { | |
| 38 | + | client := &loopingToolClient{} | |
| 39 | + | agent := NewAgent(client, "model", "system", Registry{"echo": &recordingTool{}}) | |
| 40 | + | agent.MaxSteps = 2 | |
| 41 | + | ||
| 42 | + | err := agent.Prompt(context.Background(), "keep using tools", nil, nil) | |
| 43 | + | if !errors.Is(err, ErrMaxSteps) { | |
| 44 | + | t.Fatalf("prompt error = %v, want ErrMaxSteps", err) | |
| 45 | + | } | |
| 46 | + | if got := atomic.LoadInt32(&client.calls); got != 2 { | |
| 47 | + | t.Fatalf("provider calls = %d, want 2", got) | |
| 48 | + | } | |
| 49 | + | } | |
M packages/provider/amazon_bedrock.go +14 -18
| @@ -585,28 +585,24 @@ func (c *bedrockClient) Stream(ctx context.Context, req Request) (<-chan Event, | |||
| 585 | 585 | } | |
| 586 | 586 | modelID := resolveBedrockInferenceProfileID(req.Model, c.region) | |
| 587 | 587 | url := c.baseURL + "/model/" + modelID + "/converse-stream" | |
| 588 | - | httpReq, err := http.NewRequestWithContext(ctx, "POST", url, bytes.NewReader(body)) | |
| 589 | - | if err != nil { | |
| 590 | - | return nil, err | |
| 588 | + | if c.bearerToken == "" && c.sigv4 == nil { | |
| 589 | + | return nil, fmt.Errorf("bedrock: no auth configured") | |
| 591 | 590 | } | |
| 592 | - | httpReq.Header.Set("content-type", "application/json") | |
| 593 | - | httpReq.Header.Set("accept", "application/vnd.amazon.eventstream") | |
| 594 | - | if c.bearerToken != "" { | |
| 595 | - | httpReq.Header.Set("authorization", "Bearer "+c.bearerToken) | |
| 596 | - | } else if c.sigv4 != nil { | |
| 597 | - | if err := signSigV4(httpReq, body, "bedrock", c.region, c.sigv4, time.Now().UTC()); err != nil { | |
| 591 | + | newRequest := func() (*http.Request, error) { | |
| 592 | + | httpReq, err := http.NewRequestWithContext(ctx, "POST", url, bytes.NewReader(body)) | |
| 593 | + | if err != nil { | |
| 594 | + | return nil, err | |
| 595 | + | } | |
| 596 | + | httpReq.Header.Set("content-type", "application/json") | |
| 597 | + | httpReq.Header.Set("accept", "application/vnd.amazon.eventstream") | |
| 598 | + | if c.bearerToken != "" { | |
| 599 | + | httpReq.Header.Set("authorization", "Bearer "+c.bearerToken) | |
| 600 | + | } else if err := signSigV4(httpReq, body, "bedrock", c.region, c.sigv4, time.Now().UTC()); err != nil { | |
| 598 | 601 | return nil, fmt.Errorf("bedrock: sigv4 sign: %w", err) | |
| 599 | 602 | } | |
| 600 | - | } else { | |
| 601 | - | return nil, fmt.Errorf("bedrock: no auth configured") | |
| 602 | - | } | |
| 603 | - | if err := ctx.Err(); err != nil { | |
| 604 | - | return nil, err | |
| 605 | - | } | |
| 606 | - | if req.Lifecycle != nil { | |
| 607 | - | req.Lifecycle.RequestAttempt(1, 1) | |
| 603 | + | return httpReq, nil | |
| 608 | 604 | } | |
| 609 | - | resp, err := c.http.Do(httpReq) | |
| 605 | + | resp, err := doStreamWithRetry(ctx, c.http, newRequest, req.Lifecycle) | |
| 610 | 606 | if err != nil { | |
| 611 | 607 | return nil, fmt.Errorf("bedrock: %w", err) | |
| 612 | 608 | } | |
M packages/provider/amazon_bedrock_test.go +31 -0
| @@ -2,13 +2,44 @@ package provider | |||
| 2 | 2 | ||
| 3 | 3 | import ( | |
| 4 | 4 | "bytes" | |
| 5 | + | "context" | |
| 5 | 6 | "encoding/json" | |
| 6 | 7 | "net/http" | |
| 8 | + | "net/http/httptest" | |
| 7 | 9 | "strings" | |
| 10 | + | "sync/atomic" | |
| 8 | 11 | "testing" | |
| 9 | 12 | "time" | |
| 10 | 13 | ) | |
| 11 | 14 | ||
| 15 | + | func TestBedrockRetriesTransientRequestOpenFailures(t *testing.T) { | |
| 16 | + | var attempts atomic.Int32 | |
| 17 | + | server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { | |
| 18 | + | if attempts.Add(1) < 5 { | |
| 19 | + | http.Error(w, "temporarily unavailable", http.StatusServiceUnavailable) | |
| 20 | + | return | |
| 21 | + | } | |
| 22 | + | w.WriteHeader(http.StatusOK) | |
| 23 | + | })) | |
| 24 | + | t.Cleanup(server.Close) | |
| 25 | + | ||
| 26 | + | client := &bedrockClient{ | |
| 27 | + | bearerToken: "synthetic-token", | |
| 28 | + | region: "us-east-1", | |
| 29 | + | baseURL: server.URL, | |
| 30 | + | http: server.Client(), | |
| 31 | + | } | |
| 32 | + | events, err := client.Stream(context.Background(), Request{Model: "anthropic.claude-sonnet-4-5"}) | |
| 33 | + | if err != nil { | |
| 34 | + | t.Fatal(err) | |
| 35 | + | } | |
| 36 | + | for range events { | |
| 37 | + | } | |
| 38 | + | if got := attempts.Load(); got != 5 { | |
| 39 | + | t.Fatalf("request attempts = %d, want initial request plus four retries", got) | |
| 40 | + | } | |
| 41 | + | } | |
| 42 | + | ||
| 12 | 43 | // TestSigV4Known checks our signer against AWS's published test vector | |
| 13 | 44 | // from the SigV4 documentation. The expected Authorization header below | |
| 14 | 45 | // is taken from "get-vanilla" in the suite (adapted: we use POST with | |
M packages/provider/retry.go +12 -15
| @@ -12,13 +12,10 @@ import ( | |||
| 12 | 12 | "time" | |
| 13 | 13 | ) | |
| 14 | 14 | ||
| 15 | - | // streamRetryAttempts is the maximum number of silent retries for | |
| 16 | - | // transient connect failures before the streaming request is opened. | |
| 17 | - | // 2 retries (so up to 3 total attempts) covers the common case where | |
| 18 | - | // the upstream provider's edge proxy briefly returns 502/503/504 or | |
| 19 | - | // resets the connection before headers, without making the user wait | |
| 20 | - | // noticeably long if the outage is real. | |
| 21 | - | const streamRetryAttempts = 2 | |
| 15 | + | // requestMaxRetries is the maximum number of retries for transient connect or | |
| 16 | + | // HTTP failures before a streaming response opens. It matches Codex's default | |
| 17 | + | // request retry guard and remains separate from agent-level stream reconnects. | |
| 18 | + | const requestMaxRetries = 4 | |
| 22 | 19 | ||
| 23 | 20 | // maxServerRetryDelay caps how long we are willing to honor a | |
| 24 | 21 | // server-requested Retry-After delay. When a provider asks for a | |
| @@ -27,9 +24,9 @@ const streamRetryAttempts = 2 | |||
| 27 | 24 | const maxServerRetryDelay = 60 * time.Second | |
| 28 | 25 | ||
| 29 | 26 | // streamRetryBackoff returns the wait duration before retry attempt n | |
| 30 | - | // (1-based). Short, fixed backoff: 250ms, then 750ms. Anything longer | |
| 31 | - | // would feel like the agent froze; anything shorter starts hammering | |
| 32 | - | // the provider when it's actually struggling. | |
| 27 | + | // (1-based). Short, fixed backoff: 250ms, then 750ms for later retries. | |
| 28 | + | // Anything longer would feel like the agent froze; anything shorter starts | |
| 29 | + | // hammering the provider when it's actually struggling. | |
| 33 | 30 | func streamRetryBackoff(attempt int) time.Duration { | |
| 34 | 31 | switch attempt { | |
| 35 | 32 | case 1: | |
| @@ -40,7 +37,7 @@ func streamRetryBackoff(attempt int) time.Duration { | |||
| 40 | 37 | } | |
| 41 | 38 | ||
| 42 | 39 | // doStreamWithRetry performs an HTTP request that begins a streaming | |
| 43 | - | // response, with up to streamRetryAttempts silent retries for transient | |
| 40 | + | // response, with up to requestMaxRetries silent retries for transient | |
| 44 | 41 | // connect failures. Successful responses (status 200) and non-transient | |
| 45 | 42 | // failures (4xx, malformed bodies, etc.) are returned immediately. | |
| 46 | 43 | // | |
| @@ -66,8 +63,8 @@ func streamRetryBackoff(attempt int) time.Duration { | |||
| 66 | 63 | func doStreamWithRetry(ctx context.Context, client *http.Client, newReq func() (*http.Request, error), lifecycle RequestLifecycle) (*http.Response, error) { | |
| 67 | 64 | var lastErr error | |
| 68 | 65 | serverDelay := time.Duration(-1) // <0: use the default fixed backoff | |
| 69 | - | maxAttempts := streamRetryAttempts + 1 | |
| 70 | - | for attempt := 0; attempt <= streamRetryAttempts; attempt++ { | |
| 66 | + | maxAttempts := requestMaxRetries + 1 | |
| 67 | + | for attempt := 0; attempt <= requestMaxRetries; attempt++ { | |
| 71 | 68 | if attempt > 0 { | |
| 72 | 69 | delay := streamRetryBackoff(attempt) | |
| 73 | 70 | if serverDelay >= 0 { | |
| @@ -106,7 +103,7 @@ func doStreamWithRetry(ctx context.Context, client *http.Client, newReq func() ( | |||
| 106 | 103 | return nil, err | |
| 107 | 104 | } | |
| 108 | 105 | transient := isTransientConnectError(err) | |
| 109 | - | reportRequestFailure(lifecycle, attempt+1, maxAttempts, requestFailureReasonForError(err), !transient || attempt == streamRetryAttempts) | |
| 106 | + | reportRequestFailure(lifecycle, attempt+1, maxAttempts, requestFailureReasonForError(err), !transient || attempt == requestMaxRetries) | |
| 110 | 107 | if !transient { | |
| 111 | 108 | return nil, err | |
| 112 | 109 | } | |
| @@ -129,7 +126,7 @@ func doStreamWithRetry(ctx context.Context, client *http.Client, newReq func() ( | |||
| 129 | 126 | return synthesizeResponse(resp.StatusCode, body), nil | |
| 130 | 127 | } | |
| 131 | 128 | lastErr = &transientHTTPError{Status: resp.StatusCode, Body: trimmed} | |
| 132 | - | if attempt == streamRetryAttempts { | |
| 129 | + | if attempt == requestMaxRetries { | |
| 133 | 130 | reportRequestFailure(lifecycle, attempt+1, maxAttempts, reason, true) | |
| 134 | 131 | // Wrap as a real *http.Response shape the caller | |
| 135 | 132 | // expects so it formats the error identically to a | |
M packages/provider/retry_test.go +17 -16
| @@ -19,7 +19,7 @@ func TestOpenAIStreamRetriesOn503(t *testing.T) { | |||
| 19 | 19 | var attempts atomic.Int32 | |
| 20 | 20 | srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { | |
| 21 | 21 | n := attempts.Add(1) | |
| 22 | - | if n < 3 { | |
| 22 | + | if n < 5 { | |
| 23 | 23 | w.WriteHeader(http.StatusServiceUnavailable) | |
| 24 | 24 | _, _ = w.Write([]byte("upstream connect error")) | |
| 25 | 25 | return | |
| @@ -53,14 +53,14 @@ func TestOpenAIStreamRetriesOn503(t *testing.T) { | |||
| 53 | 53 | if gotText != "ok" { | |
| 54 | 54 | t.Fatalf("text=%q", gotText) | |
| 55 | 55 | } | |
| 56 | - | if got := attempts.Load(); got != 3 { | |
| 57 | - | t.Fatalf("attempts=%d want 3", got) | |
| 56 | + | if got := attempts.Load(); got != 5 { | |
| 57 | + | t.Fatalf("attempts=%d want 5", got) | |
| 58 | 58 | } | |
| 59 | - | if got := lifecycle.attempts; len(got) != 3 || got[0] != (lifecycleAttempt{attempt: 1, max: 3}) || got[2] != (lifecycleAttempt{attempt: 3, max: 3}) { | |
| 60 | - | t.Fatalf("lifecycle attempts = %#v, want 1 through 3", got) | |
| 59 | + | if got := lifecycle.attempts; len(got) != 5 || got[0] != (lifecycleAttempt{attempt: 1, max: 5}) || got[4] != (lifecycleAttempt{attempt: 5, max: 5}) { | |
| 60 | + | t.Fatalf("lifecycle attempts = %#v, want 1 through 5", got) | |
| 61 | 61 | } | |
| 62 | - | if got := lifecycle.retries; len(got) != 2 || got[0].attempt != 2 || got[1].attempt != 3 { | |
| 63 | - | t.Fatalf("lifecycle retries = %#v, want retries before attempts 2 and 3", got) | |
| 62 | + | if got := lifecycle.retries; len(got) != 4 || got[0].attempt != 2 || got[3].attempt != 5 { | |
| 63 | + | t.Fatalf("lifecycle retries = %#v, want retries before attempts 2 through 5", got) | |
| 64 | 64 | } | |
| 65 | 65 | } | |
| 66 | 66 | ||
| @@ -85,8 +85,8 @@ func TestOpenAIStreamSurfaces503AfterRetriesExhausted(t *testing.T) { | |||
| 85 | 85 | if !strings.Contains(err.Error(), "503") { | |
| 86 | 86 | t.Fatalf("error %q should mention 503", err) | |
| 87 | 87 | } | |
| 88 | - | if got := attempts.Load(); got != streamRetryAttempts+1 { | |
| 89 | - | t.Fatalf("attempts=%d want %d", got, streamRetryAttempts+1) | |
| 88 | + | if got := attempts.Load(); got != requestMaxRetries+1 { | |
| 89 | + | t.Fatalf("attempts=%d want %d", got, requestMaxRetries+1) | |
| 90 | 90 | } | |
| 91 | 91 | } | |
| 92 | 92 | ||
| @@ -273,13 +273,14 @@ func TestDoStreamWithRetryReportsLifecycle(t *testing.T) { | |||
| 273 | 273 | } | |
| 274 | 274 | resp.Body.Close() | |
| 275 | 275 | ||
| 276 | - | if got := lifecycle.attempts; len(got) != 2 || got[0] != (lifecycleAttempt{attempt: 1, max: 3}) || got[1] != (lifecycleAttempt{attempt: 2, max: 3}) { | |
| 277 | - | t.Fatalf("attempts = %#v, want attempts 1 and 2 of 3", got) | |
| 276 | + | maxAttempts := requestMaxRetries + 1 | |
| 277 | + | if got := lifecycle.attempts; len(got) != 2 || got[0] != (lifecycleAttempt{attempt: 1, max: maxAttempts}) || got[1] != (lifecycleAttempt{attempt: 2, max: maxAttempts}) { | |
| 278 | + | t.Fatalf("attempts = %#v, want attempts 1 and 2 of %d", got, maxAttempts) | |
| 278 | 279 | } | |
| 279 | - | if got := lifecycle.retries; len(got) != 1 || got[0].attempt != 2 || got[0].max != 3 || got[0].delay != 250*time.Millisecond { | |
| 280 | - | t.Fatalf("retries = %#v, want attempt 2 of 3 after 250ms", got) | |
| 280 | + | if got := lifecycle.retries; len(got) != 1 || got[0].attempt != 2 || got[0].max != maxAttempts || got[0].delay != 250*time.Millisecond { | |
| 281 | + | t.Fatalf("retries = %#v, want attempt 2 of %d after 250ms", got, maxAttempts) | |
| 281 | 282 | } | |
| 282 | - | if got := lifecycle.failures; len(got) != 1 || got[0] != (lifecycleFailure{attempt: 1, max: 3, reason: RequestFailureOverload}) { | |
| 283 | + | if got := lifecycle.failures; len(got) != 1 || got[0] != (lifecycleFailure{attempt: 1, max: maxAttempts, reason: RequestFailureOverload}) { | |
| 283 | 284 | t.Fatalf("failures = %#v, want one non-terminal overload", got) | |
| 284 | 285 | } | |
| 285 | 286 | } | |
| @@ -302,7 +303,7 @@ func TestDoStreamWithRetryReportsTerminalClientFailure(t *testing.T) { | |||
| 302 | 303 | if resp.StatusCode != http.StatusUnauthorized { | |
| 303 | 304 | t.Fatalf("status = %d, want 401", resp.StatusCode) | |
| 304 | 305 | } | |
| 305 | - | if got := lifecycle.failures; len(got) != 1 || got[0] != (lifecycleFailure{attempt: 1, max: 3, reason: RequestFailureClient, terminal: true}) { | |
| 306 | + | if got := lifecycle.failures; len(got) != 1 || got[0] != (lifecycleFailure{attempt: 1, max: requestMaxRetries + 1, reason: RequestFailureClient, terminal: true}) { | |
| 306 | 307 | t.Fatalf("failures = %#v, want one terminal client failure", got) | |
| 307 | 308 | } | |
| 308 | 309 | if len(lifecycle.retries) != 0 { | |
| @@ -329,7 +330,7 @@ func TestDoStreamWithRetryReportsTerminalQuota(t *testing.T) { | |||
| 329 | 330 | if resp.StatusCode != http.StatusTooManyRequests { | |
| 330 | 331 | t.Fatalf("status = %d, want 429", resp.StatusCode) | |
| 331 | 332 | } | |
| 332 | - | if got := lifecycle.failures; len(got) != 1 || got[0] != (lifecycleFailure{attempt: 1, max: 3, reason: RequestFailureQuota, terminal: true}) { | |
| 333 | + | if got := lifecycle.failures; len(got) != 1 || got[0] != (lifecycleFailure{attempt: 1, max: requestMaxRetries + 1, reason: RequestFailureQuota, terminal: true}) { | |
| 333 | 334 | t.Fatalf("failures = %#v, want one terminal quota failure", got) | |
| 334 | 335 | } | |
| 335 | 336 | } | |
A packages/tui/string_spinner.go +32 -0
| @@ -0,0 +1,32 @@ | |||
| 1 | + | package tui | |
| 2 | + | ||
| 3 | + | import "time" | |
| 4 | + | ||
| 5 | + | // StringSpinner selects one text frame for each configured interval. It owns | |
| 6 | + | // no ticker or goroutine; callers render it from their existing UI clock. | |
| 7 | + | type StringSpinner struct { | |
| 8 | + | frames []string | |
| 9 | + | interval time.Duration | |
| 10 | + | } | |
| 11 | + | ||
| 12 | + | // NewStringSpinner creates a reusable text animation. Empty frames render an | |
| 13 | + | // empty string, and a non-positive interval keeps the first frame stable. | |
| 14 | + | func NewStringSpinner(frames []string, interval time.Duration) StringSpinner { | |
| 15 | + | return StringSpinner{ | |
| 16 | + | frames: append([]string(nil), frames...), | |
| 17 | + | interval: interval, | |
| 18 | + | } | |
| 19 | + | } | |
| 20 | + | ||
| 21 | + | // FrameAt returns the frame for now relative to startedAt. Times before the | |
| 22 | + | // start clamp to the first frame. | |
| 23 | + | func (s StringSpinner) FrameAt(startedAt, now time.Time) string { | |
| 24 | + | if len(s.frames) == 0 { | |
| 25 | + | return "" | |
| 26 | + | } | |
| 27 | + | if s.interval <= 0 || startedAt.IsZero() || now.IsZero() || now.Before(startedAt) { | |
| 28 | + | return s.frames[0] | |
| 29 | + | } | |
| 30 | + | index := int(now.Sub(startedAt)/s.interval) % len(s.frames) | |
| 31 | + | return s.frames[index] | |
| 32 | + | } | |
A packages/tui/string_spinner_test.go +28 -0
| @@ -0,0 +1,28 @@ | |||
| 1 | + | package tui | |
| 2 | + | ||
| 3 | + | import ( | |
| 4 | + | "testing" | |
| 5 | + | "time" | |
| 6 | + | ) | |
| 7 | + | ||
| 8 | + | func TestStringSpinnerAdvancesAtConfiguredInterval(t *testing.T) { | |
| 9 | + | spinner := NewStringSpinner([]string{"Zzzz", "zZzz", "zzZz", "zzzZ"}, time.Second) | |
| 10 | + | started := time.Date(2026, time.August, 13, 12, 0, 0, 0, time.UTC) | |
| 11 | + | ||
| 12 | + | tests := []struct { | |
| 13 | + | elapsed time.Duration | |
| 14 | + | want string | |
| 15 | + | }{ | |
| 16 | + | {elapsed: 0, want: "Zzzz"}, | |
| 17 | + | {elapsed: 999 * time.Millisecond, want: "Zzzz"}, | |
| 18 | + | {elapsed: time.Second, want: "zZzz"}, | |
| 19 | + | {elapsed: 2 * time.Second, want: "zzZz"}, | |
| 20 | + | {elapsed: 3 * time.Second, want: "zzzZ"}, | |
| 21 | + | {elapsed: 4 * time.Second, want: "Zzzz"}, | |
| 22 | + | } | |
| 23 | + | for _, test := range tests { | |
| 24 | + | if got := spinner.FrameAt(started, started.Add(test.elapsed)); got != test.want { | |
| 25 | + | t.Fatalf("FrameAt(%s) = %q, want %q", test.elapsed, got, test.want) | |
| 26 | + | } | |
| 27 | + | } | |
| 28 | + | } | |