perf(anthropic): fix cost double-count, tighten caching, correct catalog

The status-bar was showing 2x the real cost. Anthropic's SSE stream
sends the full cumulative usage payload on both message_start AND
message_delta, and our code was summing them with += on each. Cache
tokens, the biggest cost component on multi-turn sessions, were
therefore counted twice on every single API call.

Fix: assign instead of accumulate within one Stream() invocation.
Cross-call accumulation still happens correctly in
core.CostTracker.Add(). Verified end-to-end: a truly fresh "read
sample.ts on desktop" session that used to report $0.15 now reports
$0.07 with the same cache-hit rate.

While chasing that, audited and corrected the rest of the request
pipeline so the cache actually hits cleanly.

Provider layer (internal/provider/anthropic.go):
  - cache_control on the Claude Code identity line (was uncached),
    giving Anthropic a first stable checkpoint independent of the
    user system prompt. Turns a cold start from R=0 into R>0 for
    any subsequent fresh session within the cache TTL.
  - tool_result blocks go in their OWN new user message instead of
    merging into the preceding user message. Merging was mutating
    the prior user message's content array between turns, busting
    byte-identical prefix match in Anthropic's cache.
  - tagLastUserCache: exactly one cache_control on the last user
    message (was two), so identity + sysprompt + last-tool +
    last-user fits Anthropic's 4-breakpoint budget exactly.
  - user-agent dropped its "(external, cli)" suffix to match the
    canonical Claude Code string exactly.
  - ZOT_DEBUG_ANTHROPIC=<path> env hook appends each outgoing
    request body (one JSON object per line) to that file. Off by
    default; for debugging cache / cost issues in the field.
  - Usage field handling now correctly assigns the latest value
    from each SSE event instead of summing.

Core (internal/core/tool.go):
  - Registry.Specs() now sorts tools alphabetically. Go map
    iteration order is randomized per call; randomized tool arrays
    were breaking Anthropic's byte-level prefix match on every
    single call within a session.

System prompt (internal/agent/systemprompt.go):
  - Restored a substantial default prompt with structured tools +
    operating guidelines sections. The earlier aggressive trim
    dropped us under Anthropic's 1024-token minimum cacheable
    prefix floor: prefixes below 1024 tokens are silently NOT
    cached by Anthropic, so every fresh session started cold with
    R=0 no matter what else we did.
  - Current default ~1040 tokens on its own; with identity and
    tools it's ~1400, comfortably above the 1024 floor.
  - --system-prompt, --append-system-prompt, and
    $ZOT_HOME/SYSTEM.md escape hatches all still work and take
    precedence.

Model catalog (internal/provider/models.go):
  - claude-opus-4-5: 1M ctx / 128k max -> 200k ctx / 64k max. I had
    over-extrapolated; 1M context is a 4.6+ feature.
  - gpt-5.4: 400k -> 272k. Canonical value on both the OpenAI
    direct API and the ChatGPT Codex OAuth backend.
  - gpt-5.1, gpt-5.2, gpt-5.3, gpt-5.4-mini: pinned to 272k.
    OpenAI advertises 400k on direct and Codex caps at 272k. zot
    serves both from one catalog row per id, so we pin to the
    smaller number to keep the context-usage meter honest under
    subscription auth. Direct-API users see a conservative estimate
    instead of an inflated one.

README:
  - Tiny capitalization touch-up on the opening line.

commit: f3716876541e422db8e9863fb48e05cffe789271
author: patriceckhart · 6mo ago · 2026-04-19 16:57 UTC
parent: 05d0df9
signature: unsigned
refs: tag: v0.0.41

5 files · +177 -69

M README.md +1 -1


@@ -2,7 +2,7 @@
2	2
3	3		# zot
4	4
5		-	yet another coding agent harness, lightweight and written (vibe-slopped) in go.
	5	+	Yet another coding agent harness, lightweight and written (vibe-slopped) in go.
6	6
7	7		- one static binary.
8	8		- two providers atm (anthropic, openai/codex).


@@ -2,7 +2,7 @@
2			2
3		# zot	3		# zot
4			4
5	-	yet another coding agent harness, lightweight and written (vibe-slopped) in go.	5	+	Yet another coding agent harness, lightweight and written (vibe-slopped) in go.
6			6
7		- one static binary.	7		- one static binary.
8		- two providers atm (anthropic, openai/codex).	8		- two providers atm (anthropic, openai/codex).

M internal/agent/systemprompt.go +60 -17


@@ -6,10 +6,10 @@ import (
6	6		"time"
7	7		)
8	8
9		-	// ToolSummary is a name+one-line description. Kept for backwards
10		-	// compatibility with callers that still pass tool summaries in; the
11		-	// built-in system prompt no longer lists tools by name, since the
12		-	// tool schemas themselves already reach the model.
	9	+	// ToolSummary is a name+one-line description used when rendering the
	10	+	// "available tools" section of the system prompt. Passed from the
	11	+	// resolved registry so the prompt can list every tool the agent has,
	12	+	// including any extension-contributed ones.
13	13		type ToolSummary struct {
14	14		Name string
15	15		Description string

@@ -26,21 +26,25 @@ type SystemPromptOpts struct {
26	26
27	27		// BuildSystemPrompt constructs the system prompt.
28	28		//
29		-	// Design note: the prompt is intentionally tiny. Every byte here
30		-	// is re-sent on every request (cached after the first, but still
31		-	// counts toward cache-write on turn 1 and live context throughout).
32		-	// We avoid:
	29	+	// Design note on size: the prompt is sized deliberately. Anthropic's
	30	+	// prompt cache has a 1024-token minimum for the cached prefix on
	31	+	// Opus-tier models; anything smaller is NOT cached. An under-1024
	32	+	// prompt loses every fresh-session turn to "R=0" because nothing
	33	+	// about the prefix persists across invocations. Ours sits comfortably
	34	+	// above the floor so the identity + tools + this template together
	35	+	// are cached Anthropic-side across every zot invocation that shares
	36	+	// the same tool set.
33	37		//
34		-	// - Listing the tool names and descriptions (the provider sends
35		-	// the tool schemas separately; duplicating them costs tokens
36		-	// for zero benefit, the model already sees the tools).
37		-	// - Repeating generic coding-assistant advice the frontier models
38		-	// already internalise ("always read before editing", "prefer
39		-	// minimal diffs", "don't apologize"). These were free tokens
40		-	// on older models; they are pure overhead now.
	38	+	// What stays in: the identity + the tools section + concrete
	39	+	// operating guidelines that bias behaviour in ways the raw tool
	40	+	// schemas don't capture (read-before-edit, exact-match uniqueness,
	41	+	// non-interactive shell, show-don't-tell summaries). What stays out:
	42	+	// trivia and disclaimers the model already internalises.
41	43		//
42		-	// Anything the user explicitly needs can still be added via
	44	+	// Anything the user needs beyond that can be added via
43	45		// --system-prompt, --append-system-prompt, or $ZOT_HOME/SYSTEM.md.
	46	+	// A user-provided Custom fully replaces the default; --append-system-
	47	+	// prompt is additive.
44	48		func BuildSystemPrompt(o SystemPromptOpts) string {
45	49		if o.Now.IsZero() {
46	50		o.Now = time.Now()

@@ -57,6 +61,10 @@ func BuildSystemPrompt(o SystemPromptOpts) string {
57	61		sb.WriteString(o.Custom)
58	62		} else {
59	63		sb.WriteString(defaultIdentity)
	64	+	sb.WriteString("\n\n")
	65	+	sb.WriteString(renderToolsSection(o.Tools))
	66	+	sb.WriteString("\n")
	67	+	sb.WriteString(defaultGuidelines)
60	68		}
61	69
62	70		for _, a := range o.Append {

@@ -71,4 +79,39 @@ func BuildSystemPrompt(o SystemPromptOpts) string {
71	79		return sb.String()
72	80		}
73	81
74		-	const defaultIdentity = `You are zot, a lightweight terminal coding agent. Be concise, act on the user's request directly, and reply with a short summary when done.`
	82	+	// renderToolsSection lists tool names + one-line descriptions. The
	83	+	// duplication against the provider's tools array is deliberate: a
	84	+	// natural-language mention of each tool name in the system prompt
	85	+	// improves reliability of tool invocation on first-turn requests,
	86	+	// and the extra tokens help cross Anthropic's 1024-token cache floor.
	87	+	func renderToolsSection(tools []ToolSummary) string {
	88	+	if len(tools) == 0 {
	89	+	return "No tools are available in this session. Reply in plain text."
	90	+	}
	91	+	var sb strings.Builder
	92	+	sb.WriteString("You have the following tools available:\n")
	93	+	for _, t := range tools {
	94	+	fmt.Fprintf(&sb, "- %s: %s\n", t.Name, t.Description)
	95	+	}
	96	+	return sb.String()
	97	+	}
	98	+
	99	+	const defaultIdentity = `You are zot, a lightweight terminal coding agent. You help a developer by reading files, writing files, editing files, running shell commands, and calling any extension tools that are available in this session.
	100	+
	101	+	You operate inside a terminal session. Your output is rendered in a TUI that understands markdown for prose and plain text for tool-output blocks. Use markdown for explanations; let tool calls speak for themselves rather than narrating them in prose before you invoke them. Act first, then summarise what you did.
	102	+
	103	+	You are concise by default. Users running a terminal agent expect short, direct answers and precise edits. Do not apologise, do not hedge, and do not explain what you are about to do in multiple paragraphs before doing it. One short sentence of intent, then the action, then (when finished) a short recap of what changed.
	104	+
	105	+	You are careful with other people's machines. You never run destructive commands without being explicitly asked. You prefer read-only probing (` + "`ls`, `cat`, `grep`, `git status`, `go vet`, dry-run modes" + `) before making changes, and you always read a file before editing it so your exact-match replacements actually match.`
	106	+
	107	+	const defaultGuidelines = `Operating guidelines:
	108	+
	109	+	- Prefer the ` + "`edit`" + ` tool over ` + "`write`" + ` for existing files. ` + "`edit`" + ` preserves the parts of the file you are not changing, which avoids accidental deletions and keeps diffs reviewable. Use ` + "`write`" + ` only for brand new files or when you genuinely intend to overwrite the entire contents.
	110	+	- Always read a file before editing it. Your edits use exact-match text replacement; without reading you cannot know the exact bytes (whitespace, quote style, trailing newline) that the file contains. Do not guess.
	111	+	- Each ` + "`oldText`" + ` in an edit must appear exactly once in the target file. If a substring you want to replace appears multiple times, widen the context (include a few surrounding lines) until the match is unique. Do not try to replace several occurrences with the same edit.
	112	+	- Before running a shell command with ` + "`bash`" + `, explain what the command will do in one short sentence. Keep the explanation under ten words when possible. Mention side effects (network calls, file writes, process kills) so the user can stop you if needed.
	113	+	- Keep shell commands non-interactive. Pass ` + "`-y`, `--yes`, or `--non-interactive`" + ` flags where relevant. Pipe ` + "`yes`" + ` into prompts that would otherwise block. Never start a long-running server or REPL that does not exit on its own; if the user asks for one, run it in a short-lived probe (` + "`curl`, `timeout 5 …`, `echo … \| …`" + `) instead.
	114	+	- Absolutely avoid destructive commands without explicit user confirmation: ` + "`rm -rf /`, `rm -rf ~`, `dd of=/dev/`, `chmod -R 777`, `git push --force`, `git reset --hard` against unstaged work, dropping database tables, truncating migrations, reformatting drives." + ` If the user's request genuinely requires such an operation, confirm before running it.
	115	+	- When unsure about a file's contents, location, or structure, inspect first (` + "`ls`, `read`, `grep`" + `) rather than guessing a path or writing speculative code. It is always cheaper to verify than to revert a wrong edit.
	116	+	- When you finish a task, reply with a short summary of what changed and any commands the user should run (tests, builds, service restarts). Do not paste the full diff back; the TUI already showed the edits. Do not re-describe what each tool call did; just name the outcome.
	117	+	- If the user's request is ambiguous in a way that could lead to a destructive or costly action, ask a single clarifying question before proceeding. If the ambiguity is minor (naming, style, placement), pick a sensible default and mention it in your summary so the user can redirect.`


@@ -6,10 +6,10 @@ import (
6		"time"	6		"time"
7		)	7		)
8			8
9	-	// ToolSummary is a name+one-line description. Kept for backwards	9	+	// ToolSummary is a name+one-line description used when rendering the
10	-	// compatibility with callers that still pass tool summaries in; the	10	+	// "available tools" section of the system prompt. Passed from the
11	-	// built-in system prompt no longer lists tools by name, since the	11	+	// resolved registry so the prompt can list every tool the agent has,
12	-	// tool schemas themselves already reach the model.	12	+	// including any extension-contributed ones.
13		type ToolSummary struct {	13		type ToolSummary struct {
14		Name string	14		Name string
15		Description string	15		Description string

@@ -26,21 +26,25 @@ type SystemPromptOpts struct {
26			26
27		// BuildSystemPrompt constructs the system prompt.	27		// BuildSystemPrompt constructs the system prompt.
28		//	28		//
29	-	// Design note: the prompt is intentionally tiny. Every byte here	29	+	// Design note on size: the prompt is sized deliberately. Anthropic's
30	-	// is re-sent on every request (cached after the first, but still	30	+	// prompt cache has a 1024-token minimum for the cached prefix on
31	-	// counts toward cache-write on turn 1 and live context throughout).	31	+	// Opus-tier models; anything smaller is NOT cached. An under-1024
32	-	// We avoid:	32	+	// prompt loses every fresh-session turn to "R=0" because nothing
			33	+	// about the prefix persists across invocations. Ours sits comfortably
			34	+	// above the floor so the identity + tools + this template together
			35	+	// are cached Anthropic-side across every zot invocation that shares
			36	+	// the same tool set.
33		//	37		//
34	-	// - Listing the tool names and descriptions (the provider sends	38	+	// What stays in: the identity + the tools section + concrete
35	-	// the tool schemas separately; duplicating them costs tokens	39	+	// operating guidelines that bias behaviour in ways the raw tool
36	-	// for zero benefit, the model already sees the tools).	40	+	// schemas don't capture (read-before-edit, exact-match uniqueness,
37	-	// - Repeating generic coding-assistant advice the frontier models	41	+	// non-interactive shell, show-don't-tell summaries). What stays out:
38	-	// already internalise ("always read before editing", "prefer	42	+	// trivia and disclaimers the model already internalises.
39	-	// minimal diffs", "don't apologize"). These were free tokens
40	-	// on older models; they are pure overhead now.
41		//	43		//
42	-	// Anything the user explicitly needs can still be added via	44	+	// Anything the user needs beyond that can be added via
43		// --system-prompt, --append-system-prompt, or $ZOT_HOME/SYSTEM.md.	45		// --system-prompt, --append-system-prompt, or $ZOT_HOME/SYSTEM.md.
			46	+	// A user-provided Custom fully replaces the default; --append-system-
			47	+	// prompt is additive.
44		func BuildSystemPrompt(o SystemPromptOpts) string {	48		func BuildSystemPrompt(o SystemPromptOpts) string {
45		if o.Now.IsZero() {	49		if o.Now.IsZero() {
46		o.Now = time.Now()	50		o.Now = time.Now()

@@ -57,6 +61,10 @@ func BuildSystemPrompt(o SystemPromptOpts) string {
57		sb.WriteString(o.Custom)	61		sb.WriteString(o.Custom)
58		} else {	62		} else {
59		sb.WriteString(defaultIdentity)	63		sb.WriteString(defaultIdentity)
			64	+	sb.WriteString("\n\n")
			65	+	sb.WriteString(renderToolsSection(o.Tools))
			66	+	sb.WriteString("\n")
			67	+	sb.WriteString(defaultGuidelines)
60		}	68		}
61			69
62		for _, a := range o.Append {	70		for _, a := range o.Append {

@@ -71,4 +79,39 @@ func BuildSystemPrompt(o SystemPromptOpts) string {
71		return sb.String()	79		return sb.String()
72		}	80		}
73			81
74	-	const defaultIdentity = `You are zot, a lightweight terminal coding agent. Be concise, act on the user's request directly, and reply with a short summary when done.`	82	+	// renderToolsSection lists tool names + one-line descriptions. The
			83	+	// duplication against the provider's tools array is deliberate: a
			84	+	// natural-language mention of each tool name in the system prompt
			85	+	// improves reliability of tool invocation on first-turn requests,
			86	+	// and the extra tokens help cross Anthropic's 1024-token cache floor.
			87	+	func renderToolsSection(tools []ToolSummary) string {
			88	+	if len(tools) == 0 {
			89	+	return "No tools are available in this session. Reply in plain text."
			90	+	}
			91	+	var sb strings.Builder
			92	+	sb.WriteString("You have the following tools available:\n")
			93	+	for _, t := range tools {
			94	+	fmt.Fprintf(&sb, "- %s: %s\n", t.Name, t.Description)
			95	+	}
			96	+	return sb.String()
			97	+	}
			98	+
			99	+	const defaultIdentity = `You are zot, a lightweight terminal coding agent. You help a developer by reading files, writing files, editing files, running shell commands, and calling any extension tools that are available in this session.
			100	+
			101	+	You operate inside a terminal session. Your output is rendered in a TUI that understands markdown for prose and plain text for tool-output blocks. Use markdown for explanations; let tool calls speak for themselves rather than narrating them in prose before you invoke them. Act first, then summarise what you did.
			102	+
			103	+	You are concise by default. Users running a terminal agent expect short, direct answers and precise edits. Do not apologise, do not hedge, and do not explain what you are about to do in multiple paragraphs before doing it. One short sentence of intent, then the action, then (when finished) a short recap of what changed.
			104	+
			105	+	You are careful with other people's machines. You never run destructive commands without being explicitly asked. You prefer read-only probing (` + "`ls`, `cat`, `grep`, `git status`, `go vet`, dry-run modes" + `) before making changes, and you always read a file before editing it so your exact-match replacements actually match.`
			106	+
			107	+	const defaultGuidelines = `Operating guidelines:
			108	+
			109	+	- Prefer the ` + "`edit`" + ` tool over ` + "`write`" + ` for existing files. ` + "`edit`" + ` preserves the parts of the file you are not changing, which avoids accidental deletions and keeps diffs reviewable. Use ` + "`write`" + ` only for brand new files or when you genuinely intend to overwrite the entire contents.
			110	+	- Always read a file before editing it. Your edits use exact-match text replacement; without reading you cannot know the exact bytes (whitespace, quote style, trailing newline) that the file contains. Do not guess.
			111	+	- Each ` + "`oldText`" + ` in an edit must appear exactly once in the target file. If a substring you want to replace appears multiple times, widen the context (include a few surrounding lines) until the match is unique. Do not try to replace several occurrences with the same edit.
			112	+	- Before running a shell command with ` + "`bash`" + `, explain what the command will do in one short sentence. Keep the explanation under ten words when possible. Mention side effects (network calls, file writes, process kills) so the user can stop you if needed.
			113	+	- Keep shell commands non-interactive. Pass ` + "`-y`, `--yes`, or `--non-interactive`" + ` flags where relevant. Pipe ` + "`yes`" + ` into prompts that would otherwise block. Never start a long-running server or REPL that does not exit on its own; if the user asks for one, run it in a short-lived probe (` + "`curl`, `timeout 5 …`, `echo … \| …`" + `) instead.
			114	+	- Absolutely avoid destructive commands without explicit user confirmation: ` + "`rm -rf /`, `rm -rf ~`, `dd of=/dev/`, `chmod -R 777`, `git push --force`, `git reset --hard` against unstaged work, dropping database tables, truncating migrations, reformatting drives." + ` If the user's request genuinely requires such an operation, confirm before running it.
			115	+	- When unsure about a file's contents, location, or structure, inspect first (` + "`ls`, `read`, `grep`" + `) rather than guessing a path or writing speculative code. It is always cheaper to verify than to revert a wrong edit.
			116	+	- When you finish a task, reply with a short summary of what changed and any commands the user should run (tests, builds, service restarts). Do not paste the full diff back; the TUI already showed the edits. Do not re-describe what each tool call did; just name the outcome.
			117	+	- If the user's request is ambiguous in a way that could lead to a destructive or costly action, ask a single clarifying question before proceeding. If the ambiguity is minor (naming, style, placement), pick a sensible default and mention it in your summary so the user can redirect.`

M internal/core/tool.go +13 -1


@@ -7,6 +7,7 @@ import (
7	7		"context"
8	8		"encoding/json"
9	9		"fmt"
	10	+	"sort"
10	11
11	12		"github.com/patriceckhart/zot/internal/provider"
12	13		)

@@ -47,9 +48,20 @@ func NewRegistry(tools ...Tool) Registry {
47	48		}
48	49
49	50		// Specs returns the tool definitions to advertise to the LLM.
	51	+	// Sorted by tool name so the order is stable across requests. This
	52	+	// is load-bearing for provider-side prompt caching: providers
	53	+	// prefix-match tool definitions, and Go's map iteration order is
	54	+	// randomized per call, which would otherwise bust the cache every
	55	+	// single turn.
50	56		func (r Registry) Specs() []provider.Tool {
	57	+	names := make([]string, 0, len(r))
	58	+	for name := range r {
	59	+	names = append(names, name)
	60	+	}
	61	+	sort.Strings(names)
51	62		out := make([]provider.Tool, 0, len(r))
52		-	for _, t := range r {
	63	+	for _, name := range names {
	64	+	t := r[name]
53	65		out = append(out, provider.Tool{
54	66		Name: t.Name(),
55	67		Description: t.Description(),

 	"context"
 	"encoding/json"
 	"fmt"
+	"sort"
 	"github.com/patriceckhart/zot/internal/provider"
 )
 }
 // Specs returns the tool definitions to advertise to the LLM.
+// Sorted by tool name so the order is stable across requests. This
+// is load-bearing for provider-side prompt caching: providers
+// prefix-match tool definitions, and Go's map iteration order is
+// randomized per call, which would otherwise bust the cache every
+// single turn.
 func (r Registry) Specs() []provider.Tool {
+	names := make([]string, 0, len(r))
+	for name := range r {
+		names = append(names, name)
+	}
+	sort.Strings(names)
 	out := make([]provider.Tool, 0, len(r))
-	for _, t := range r {
+	for _, name := range names {
+		t := r[name]
 		out = append(out, provider.Tool{
 			Name:        t.Name(),
 			Description: t.Description(),

M internal/provider/anthropic.go +82 -44


@@ -8,6 +8,7 @@ import (
8	8		"fmt"
9	9		"io"
10	10		"net/http"
	11	+	"os"
11	12		"strings"
12	13		"time"
13	14		)

@@ -186,14 +187,24 @@ func (c anthropicClient) buildRequest(req Request) (anthRequest, error) {
186	187		// Anthropic rejects them (429 rate_limit_error with zero tokens used).
187	188		//
188	189		// Cache budget: anthropic caps cache_control to 4 breakpoints per
189		-	// request. We spend them on (system prompt) + (tools tail) + (last
190		-	// two user messages). The claude-code identity line stays uncached
191		-	// because it's a few tokens and gets folded into the larger prefix
192		-	// implicitly anyway.
	190	+	// request. We spend them on:
	191	+	// 1. claude-code identity (OAuth only; stable forever)
	192	+	// 2. user system prompt (changes per-session at most)
	193	+	// 3. last tool definition (tools change rarely)
	194	+	// 4. last message block (advances every turn)
	195	+	//
	196	+	// The identity line gets its OWN cache_control so the prefix
	197	+	// [identity] is cacheable independently of the user system
	198	+	// prompt. Without that, the cache prefix starts after block 2
	199	+	// and any drift in the user prompt (e.g. the Current date
	200	+	// line flipping at midnight) invalidates everything, including
	201	+	// the 17 identity tokens we have to re-send every request
	202	+	// forever.
193	203		if c.oauthTok != "" {
194	204		out.System = []anthSystemBlock{{
195		-	Type: "text",
196		-	Text: claudeCodeIdentity,
	205	+	Type: "text",
	206	+	Text: claudeCodeIdentity,
	207	+	CacheControl: &anthCacheCtrl{Type: "ephemeral"},
197	208		}}
198	209		if req.System != "" {
199	210		out.System = append(out.System, anthSystemBlock{

@@ -237,8 +248,20 @@ func (c anthropicClient) buildRequest(req Request) (anthRequest, error) {
237	248		out.Tools[n-1].CacheControl = &anthCacheCtrl{Type: "ephemeral"}
238	249		}
239	250
240		-	// Group messages: consecutive user/tool roles into one "user" message.
241		-	// Anthropic only has roles "user" and "assistant"; tool_result blocks live in user messages.
	251	+	// Convert messages. Anthropic's wire format has only "user" and
	252	+	// "assistant" roles; tool_result blocks live inside user messages.
	253	+	//
	254	+	// CRITICAL: tool_result blocks go into their OWN new user
	255	+	// message, they are NOT merged into the preceding user message.
	256	+	// Merging would mutate the prior user message's content array
	257	+	// between turn N and turn N+1: turn N caches the prefix ending at
	258	+	// [user: "read sample.ts"], turn N+1 sends
	259	+	// [user: "read sample.ts" + tool_result=...] which is a
	260	+	// different block sequence, busting the cache prefix match.
	261	+	// Anthropic's API happily accepts consecutive user messages, and
	262	+	// emitting them separately keeps each message bit-stable across
	263	+	// turns, so the cache prefix matches for the entire history up
	264	+	// to the newest block.
242	265		for _, msg := range req.Messages {
243	266		renameTools := c.oauthTok != ""
244	267		switch msg.Role {

@@ -248,13 +271,10 @@ func (c anthropicClient) buildRequest(req Request) (anthRequest, error) {
248	271		Content: convertAnthContent(msg.Content, renameTools),
249	272		})
250	273		case RoleTool:
251		-	// Attach tool_result blocks to a user message; merge with prior user msg if last.
252		-	blocks := convertAnthContent(msg.Content, renameTools)
253		-	if n := len(out.Messages); n > 0 && out.Messages[n-1].Role == "user" {
254		-	out.Messages[n-1].Content = append(out.Messages[n-1].Content, blocks...)
255		-	} else {
256		-	out.Messages = append(out.Messages, anthMessage{Role: "user", Content: blocks})
257		-	}
	274	+	out.Messages = append(out.Messages, anthMessage{
	275	+	Role: "user",
	276	+	Content: convertAnthContent(msg.Content, renameTools),
	277	+	})
258	278		case RoleAssistant:
259	279		out.Messages = append(out.Messages, anthMessage{
260	280		Role: "assistant",

@@ -263,33 +283,24 @@ func (c anthropicClient) buildRequest(req Request) (anthRequest, error) {
263	283		}
264	284		}
265	285
266		-	// Mark the last two user messages with cache_control so anthropic
267		-	// caches the running conversation prefix. Combined with the system
268		-	// + tools breakpoints above this is the recommended layout for
269		-	// multi-turn caching: turn N writes a cache that turn N+1 reads,
270		-	// dropping per-turn input tokens from "system + history" down to
271		-	// just the new user message. Anthropic allows up to 4 breakpoints
272		-	// per request; we use system + tools + 2 conversation = 4.
273		-	tagUserCache(out.Messages)
	286	+	// Tag the LAST user message with cache_control. Spends the 4th
	287	+	// breakpoint. For prefixes under ~1024 tokens (Anthropic's
	288	+	// minimum cacheable block size for Opus), no cache is written.
	289	+	tagLastUserCache(out.Messages)
274	290
275	291		return out, nil
276	292		}
277	293
278		-	// tagUserCache attaches a cache_control marker to the last block of
279		-	// the most recent (and second-most-recent, if any) user message in
280		-	// msgs. The marker tells the api to checkpoint the prefix at that
281		-	// point so subsequent requests can replay everything up to and
282		-	// including that block as a cache hit.
283		-	func tagUserCache(msgs []anthMessage) {
284		-	indexes := make([]int, 0, 2)
285		-	for i := len(msgs) - 1; i >= 0 && len(indexes) < 2; i-- {
	294	+	// tagLastUserCache marks the last block of the most recent user
	295	+	// message. One marker; combined with identity + systemPrompt +
	296	+	// last-tool, spends Anthropic's 4-breakpoint budget.
	297	+	func tagLastUserCache(msgs []anthMessage) {
	298	+	for i := len(msgs) - 1; i >= 0; i-- {
286	299		if msgs[i].Role == "user" {
287		-	indexes = append(indexes, i)
	300	+	markLastBlockEphemeral(msgs[i].Content)
	301	+	return
288	302		}
289	303		}
290		-	for _, idx := range indexes {
291		-	markLastBlockEphemeral(msgs[idx].Content)
292		-	}
293	304		}
294	305
295	306		// markLastBlockEphemeral sets CacheControl on the last entry in blocks

@@ -427,6 +438,18 @@ func (c *anthropicClient) Stream(ctx context.Context, req Request) (<-chan Event
427	438		return nil, err
428	439		}
429	440
	441	+	// Optional debug dump: when $ZOT_DEBUG_ANTHROPIC is a file path
	442	+	// we append every outgoing request body to it, one JSON object
	443	+	// per line. Useful for diffing turn N vs turn N+1 to understand
	444	+	// why the cache prefix isn't matching.
	445	+	if dump := os.Getenv("ZOT_DEBUG_ANTHROPIC"); dump != "" {
	446	+	if f, derr := os.OpenFile(dump, os.O_CREATE\|os.O_WRONLY\|os.O_APPEND, 0o600); derr == nil {
	447	+	_, _ = f.Write(body)
	448	+	_, _ = f.Write([]byte{'\n'})
	449	+	_ = f.Close()
	450	+	}
	451	+	}
	452	+
430	453		httpReq, err := http.NewRequestWithContext(ctx, "POST", c.baseURL+"/v1/messages", bytes.NewReader(body))
431	454		if err != nil {
432	455		return nil, err

@@ -441,7 +464,7 @@ func (c *anthropicClient) Stream(ctx context.Context, req Request) (<-chan Event
441	464		httpReq.Header.Set("authorization", "Bearer "+c.oauthTok)
442	465		httpReq.Header.Set("anthropic-beta", "claude-code-20250219,oauth-2025-04-20,fine-grained-tool-streaming-2025-05-14")
443	466		httpReq.Header.Set("anthropic-dangerous-direct-browser-access", "true")
444		-	httpReq.Header.Set("user-agent", "claude-cli/"+claudeCodeVersion+" (external, cli)")
	467	+	httpReq.Header.Set("user-agent", "claude-cli/"+claudeCodeVersion)
445	468		httpReq.Header.Set("x-app", "cli")
446	469		// Remove x-api-key entirely by NOT setting it.
447	470		} else {

@@ -629,10 +652,14 @@ func (c anthropicClient) runStream(ctx context.Context, resp http.Response, re
629	652		} `json:"message"`
630	653		}
631	654		_ = json.Unmarshal([]byte(ev.Data), &m)
632		-	usage.InputTokens += m.Message.Usage.InputTokens
633		-	usage.OutputTokens += m.Message.Usage.OutputTokens
634		-	usage.CacheReadTokens += m.Message.Usage.CacheReadInputTokens
635		-	usage.CacheWriteTokens += m.Message.Usage.CacheCreationInputTokens
	655	+	// Anthropic sends cumulative values on message_start and
	656	+	// again on message_delta (refreshed), so assign, don't
	657	+	// accumulate. Accumulating doubles cache_creation_input
	658	+	// which can be 50-70% of cost.
	659	+	usage.InputTokens = m.Message.Usage.InputTokens
	660	+	usage.OutputTokens = m.Message.Usage.OutputTokens
	661	+	usage.CacheReadTokens = m.Message.Usage.CacheReadInputTokens
	662	+	usage.CacheWriteTokens = m.Message.Usage.CacheCreationInputTokens
636	663		case "message_delta":
637	664		var m struct {
638	665		Delta struct {

@@ -646,10 +673,21 @@ func (c anthropicClient) runStream(ctx context.Context, resp http.Response, re
646	673		} `json:"usage"`
647	674		}
648	675		_ = json.Unmarshal([]byte(ev.Data), &m)
649		-	usage.InputTokens += m.Usage.InputTokens
650		-	usage.OutputTokens += m.Usage.OutputTokens
651		-	usage.CacheReadTokens += m.Usage.CacheReadInputTokens
652		-	usage.CacheWriteTokens += m.Usage.CacheCreationInputTokens
	676	+	// Refresh usage from the latest cumulative totals
	677	+	// Anthropic provides. Only apply non-zero fields in case
	678	+	// a given delta only carries output tokens.
	679	+	if m.Usage.InputTokens > 0 {
	680	+	usage.InputTokens = m.Usage.InputTokens
	681	+	}
	682	+	if m.Usage.OutputTokens > 0 {
	683	+	usage.OutputTokens = m.Usage.OutputTokens
	684	+	}
	685	+	if m.Usage.CacheReadInputTokens > 0 {
	686	+	usage.CacheReadTokens = m.Usage.CacheReadInputTokens
	687	+	}
	688	+	if m.Usage.CacheCreationInputTokens > 0 {
	689	+	usage.CacheWriteTokens = m.Usage.CacheCreationInputTokens
	690	+	}
653	691		switch m.Delta.StopReason {
654	692		case "end_turn", "stop_sequence":
655	693		stop = StopEnd

 	"fmt"
 	"io"
 	"net/http"
+	"os"
 	"strings"
 	"time"
 )
 	// Anthropic rejects them (429 rate_limit_error with zero tokens used).
 	//
 	// Cache budget: anthropic caps cache_control to 4 breakpoints per
-	// request. We spend them on (system prompt) + (tools tail) + (last
+	// request. We spend them on:
-	// two user messages). The claude-code identity line stays uncached
+	//   1. claude-code identity (OAuth only; stable forever)
-	// because it's a few tokens and gets folded into the larger prefix
+	//   2. user system prompt   (changes per-session at most)
-	// implicitly anyway.
+	//   3. last tool definition (tools change rarely)
+	//   4. last message block   (advances every turn)
+	//
+	// The identity line gets its OWN cache_control so the prefix
+	// [identity] is cacheable independently of the user system
+	// prompt. Without that, the cache prefix starts after block 2
+	// and any drift in the user prompt (e.g. the Current date
+	// line flipping at midnight) invalidates everything, including
+	// the 17 identity tokens we have to re-send every request
+	// forever.
 	if c.oauthTok != "" {
 		out.System = []anthSystemBlock{{
-			Type: "text",
+			Type:         "text",
-			Text: claudeCodeIdentity,
+			Text:         claudeCodeIdentity,
+			CacheControl: &anthCacheCtrl{Type: "ephemeral"},
 		}}
 		if req.System != "" {
 			out.System = append(out.System, anthSystemBlock{
 		out.Tools[n-1].CacheControl = &anthCacheCtrl{Type: "ephemeral"}
 	}
-	// Group messages: consecutive user/tool roles into one "user" message.
+	// Convert messages. Anthropic's wire format has only "user" and
-	// Anthropic only has roles "user" and "assistant"; tool_result blocks live in user messages.
+	// "assistant" roles; tool_result blocks live inside user messages.
+	//
+	// CRITICAL: tool_result blocks go into their OWN new user
+	// message, they are NOT merged into the preceding user message.
+	// Merging would mutate the prior user message's content array
+	// between turn N and turn N+1: turn N caches the prefix ending at
+	// [user: "read sample.ts"], turn N+1 sends
+	// [user: "read sample.ts" + tool_result=...] which is a
+	// different block sequence, busting the cache prefix match.
+	// Anthropic's API happily accepts consecutive user messages, and
+	// emitting them separately keeps each message bit-stable across
+	// turns, so the cache prefix matches for the entire history up
+	// to the newest block.
 	for _, msg := range req.Messages {
 		renameTools := c.oauthTok != ""
 		switch msg.Role {
 				Content: convertAnthContent(msg.Content, renameTools),
 			})
 		case RoleTool:
-			// Attach tool_result blocks to a user message; merge with prior user msg if last.
+			out.Messages = append(out.Messages, anthMessage{
-			blocks := convertAnthContent(msg.Content, renameTools)
+				Role:    "user",
-			if n := len(out.Messages); n > 0 && out.Messages[n-1].Role == "user" {
+				Content: convertAnthContent(msg.Content, renameTools),
-				out.Messages[n-1].Content = append(out.Messages[n-1].Content, blocks...)
+			})
-			} else {
-				out.Messages = append(out.Messages, anthMessage{Role: "user", Content: blocks})
-			}
 		case RoleAssistant:
 			out.Messages = append(out.Messages, anthMessage{
 				Role:    "assistant",
 		}
 	}
-	// Mark the last two user messages with cache_control so anthropic
+	// Tag the LAST user message with cache_control. Spends the 4th
-	// caches the running conversation prefix. Combined with the system
+	// breakpoint. For prefixes under ~1024 tokens (Anthropic's
-	// + tools breakpoints above this is the recommended layout for
+	// minimum cacheable block size for Opus), no cache is written.
-	// multi-turn caching: turn N writes a cache that turn N+1 reads,
+	tagLastUserCache(out.Messages)
-	// dropping per-turn input tokens from "system + history" down to
-	// just the new user message. Anthropic allows up to 4 breakpoints
-	// per request; we use system + tools + 2 conversation = 4.
-	tagUserCache(out.Messages)
 	return out, nil
 }
-// tagUserCache attaches a cache_control marker to the last block of
+// tagLastUserCache marks the last block of the most recent user
-// the most recent (and second-most-recent, if any) user message in
+// message. One marker; combined with identity + systemPrompt +
-// msgs. The marker tells the api to checkpoint the prefix at that
+// last-tool, spends Anthropic's 4-breakpoint budget.
-// point so subsequent requests can replay everything up to and
+func tagLastUserCache(msgs []anthMessage) {
-// including that block as a cache hit.
+	for i := len(msgs) - 1; i >= 0; i-- {
-func tagUserCache(msgs []anthMessage) {
-	indexes := make([]int, 0, 2)
-	for i := len(msgs) - 1; i >= 0 && len(indexes) < 2; i-- {
 		if msgs[i].Role == "user" {
-			indexes = append(indexes, i)
+			markLastBlockEphemeral(msgs[i].Content)
+			return
 		}
 	}
-	for _, idx := range indexes {
-		markLastBlockEphemeral(msgs[idx].Content)
-	}
 }
 // markLastBlockEphemeral sets CacheControl on the last entry in blocks
 		return nil, err
 	}
+	// Optional debug dump: when $ZOT_DEBUG_ANTHROPIC is a file path
+	// we append every outgoing request body to it, one JSON object
+	// per line. Useful for diffing turn N vs turn N+1 to understand
+	// why the cache prefix isn't matching.
+	if dump := os.Getenv("ZOT_DEBUG_ANTHROPIC"); dump != "" {
+		if f, derr := os.OpenFile(dump, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o600); derr == nil {
+			_, _ = f.Write(body)
+			_, _ = f.Write([]byte{'\n'})
+			_ = f.Close()
+		}
+	}
 	httpReq, err := http.NewRequestWithContext(ctx, "POST", c.baseURL+"/v1/messages", bytes.NewReader(body))
 	if err != nil {
 		return nil, err
 		httpReq.Header.Set("authorization", "Bearer "+c.oauthTok)
 		httpReq.Header.Set("anthropic-beta", "claude-code-20250219,oauth-2025-04-20,fine-grained-tool-streaming-2025-05-14")
 		httpReq.Header.Set("anthropic-dangerous-direct-browser-access", "true")
-		httpReq.Header.Set("user-agent", "claude-cli/"+claudeCodeVersion+" (external, cli)")
+		httpReq.Header.Set("user-agent", "claude-cli/"+claudeCodeVersion)
 		httpReq.Header.Set("x-app", "cli")
 		// Remove x-api-key entirely by NOT setting it.
 	} else {
 					} `json:"message"`
 				}
 				_ = json.Unmarshal([]byte(ev.Data), &m)
-				usage.InputTokens += m.Message.Usage.InputTokens
+				// Anthropic sends cumulative values on message_start and
-				usage.OutputTokens += m.Message.Usage.OutputTokens
+				// again on message_delta (refreshed), so assign, don't
-				usage.CacheReadTokens += m.Message.Usage.CacheReadInputTokens
+				// accumulate. Accumulating doubles cache_creation_input
-				usage.CacheWriteTokens += m.Message.Usage.CacheCreationInputTokens
+				// which can be 50-70% of cost.
+				usage.InputTokens = m.Message.Usage.InputTokens
+				usage.OutputTokens = m.Message.Usage.OutputTokens
+				usage.CacheReadTokens = m.Message.Usage.CacheReadInputTokens
+				usage.CacheWriteTokens = m.Message.Usage.CacheCreationInputTokens
 			case "message_delta":
 				var m struct {
 					Delta struct {
 					} `json:"usage"`
 				}
 				_ = json.Unmarshal([]byte(ev.Data), &m)
-				usage.InputTokens += m.Usage.InputTokens
+				// Refresh usage from the latest cumulative totals
-				usage.OutputTokens += m.Usage.OutputTokens
+				// Anthropic provides. Only apply non-zero fields in case
-				usage.CacheReadTokens += m.Usage.CacheReadInputTokens
+				// a given delta only carries output tokens.
-				usage.CacheWriteTokens += m.Usage.CacheCreationInputTokens
+				if m.Usage.InputTokens > 0 {
+					usage.InputTokens = m.Usage.InputTokens
+				}
+				if m.Usage.OutputTokens > 0 {
+					usage.OutputTokens = m.Usage.OutputTokens
+				}
+				if m.Usage.CacheReadInputTokens > 0 {
+					usage.CacheReadTokens = m.Usage.CacheReadInputTokens
+				}
+				if m.Usage.CacheCreationInputTokens > 0 {
+					usage.CacheWriteTokens = m.Usage.CacheCreationInputTokens
+				}
 				switch m.Delta.StopReason {
 				case "end_turn", "stop_sequence":
 					stop = StopEnd

M internal/provider/models.go +21 -6


@@ -157,7 +157,9 @@ var Catalog = []Model{
157	157		// ---- Speculative: Anthropic ----
158	158		{
159	159		Provider: "anthropic", ID: "claude-opus-4-5", DisplayName: "Claude Opus 4.5",
160		-	ContextWindow: 1000000, MaxOutput: 128000, Reasoning: true,
	160	+	// 200k ctx / 64k maxOutput per Anthropic's published sizing
	161	+	// for the opus-4-5 family; the 1M context is a 4.6+ thing.
	162	+	ContextWindow: 200000, MaxOutput: 64000, Reasoning: true,
161	163		PriceInput: 5.00, PriceOutput: 25.00, PriceCacheRead: 0.50, PriceCacheWrite: 6.25,
162	164		Speculative: true,
163	165		},

@@ -181,33 +183,46 @@ var Catalog = []Model{
181	183		},
182	184
183	185		// ---- Speculative: OpenAI ----
	186	+	// Context windows on the OpenAI gpt-5.x family differ by route:
	187	+	// the direct API advertises 400k, the ChatGPT Codex OAuth backend
	188	+	// caps at 272k. zot serves both auth modes from one catalog row
	189	+	// per id, so we pin to the smaller number to keep the context-usage
	190	+	// meter honest under subscription auth. Users on the direct API
	191	+	// simply see a conservative headroom estimate.
184	192		{
185	193		Provider: "openai", ID: "gpt-5.1", DisplayName: "GPT-5.1",
186		-	ContextWindow: 400000, MaxOutput: 128000, Reasoning: true,
	194	+	ContextWindow: 272000, MaxOutput: 128000, Reasoning: true,
187	195		PriceInput: 1.25, PriceOutput: 10.00, PriceCacheRead: 0.125,
188	196		Speculative: true,
189	197		},
190	198		{
191	199		Provider: "openai", ID: "gpt-5.2", DisplayName: "GPT-5.2",
192		-	ContextWindow: 400000, MaxOutput: 128000, Reasoning: true,
	200	+	ContextWindow: 272000, MaxOutput: 128000, Reasoning: true,
193	201		PriceInput: 1.75, PriceOutput: 14.00, PriceCacheRead: 0.175,
194	202		Speculative: true,
195	203		},
196	204		{
197	205		Provider: "openai", ID: "gpt-5.3", DisplayName: "GPT-5.3",
198		-	ContextWindow: 400000, MaxOutput: 128000, Reasoning: true,
	206	+	ContextWindow: 272000, MaxOutput: 128000, Reasoning: true,
199	207		PriceInput: 1.75, PriceOutput: 14.00, PriceCacheRead: 0.175,
200	208		Speculative: true,
201	209		},
202	210		{
203	211		Provider: "openai", ID: "gpt-5.4", DisplayName: "GPT-5.4",
204		-	ContextWindow: 400000, MaxOutput: 128000, Reasoning: true,
	212	+	// ContextWindow: 272k across every route we support (OpenAI
	213	+	// direct API and the ChatGPT Codex OAuth backend).
	214	+	ContextWindow: 272000, MaxOutput: 128000, Reasoning: true,
205	215		PriceInput: 2.50, PriceOutput: 15.00, PriceCacheRead: 0.25,
206	216		Speculative: true,
207	217		},
208	218		{
209	219		Provider: "openai", ID: "gpt-5.4-mini", DisplayName: "GPT-5.4 mini",
210		-	ContextWindow: 400000, MaxOutput: 128000, Reasoning: true,
	220	+	// ContextWindow: 400k on the OpenAI direct API, 272k on the
	221	+	// ChatGPT Codex OAuth backend. We pin to the smaller Codex
	222	+	// cap so the context-usage meter is honest under subscription
	223	+	// auth; direct-API users simply see a conservative headroom
	224	+	// estimate rather than an inflated one.
	225	+	ContextWindow: 272000, MaxOutput: 128000, Reasoning: true,
211	226		PriceInput: 0.75, PriceOutput: 4.50, PriceCacheRead: 0.075,
212	227		Speculative: true,
213	228		},

 	// ---- Speculative: Anthropic ----
 	{
 		Provider: "anthropic", ID: "claude-opus-4-5", DisplayName: "Claude Opus 4.5",
-		ContextWindow: 1000000, MaxOutput: 128000, Reasoning: true,
+		// 200k ctx / 64k maxOutput per Anthropic's published sizing
+		// for the opus-4-5 family; the 1M context is a 4.6+ thing.
+		ContextWindow: 200000, MaxOutput: 64000, Reasoning: true,
 		PriceInput: 5.00, PriceOutput: 25.00, PriceCacheRead: 0.50, PriceCacheWrite: 6.25,
 		Speculative: true,
 	},
 	},
 	// ---- Speculative: OpenAI ----
+	// Context windows on the OpenAI gpt-5.x family differ by route:
+	// the direct API advertises 400k, the ChatGPT Codex OAuth backend
+	// caps at 272k. zot serves both auth modes from one catalog row
+	// per id, so we pin to the smaller number to keep the context-usage
+	// meter honest under subscription auth. Users on the direct API
+	// simply see a conservative headroom estimate.
 	{
 		Provider: "openai", ID: "gpt-5.1", DisplayName: "GPT-5.1",
-		ContextWindow: 400000, MaxOutput: 128000, Reasoning: true,
+		ContextWindow: 272000, MaxOutput: 128000, Reasoning: true,
 		PriceInput: 1.25, PriceOutput: 10.00, PriceCacheRead: 0.125,
 		Speculative: true,
 	},
 	{
 		Provider: "openai", ID: "gpt-5.2", DisplayName: "GPT-5.2",
-		ContextWindow: 400000, MaxOutput: 128000, Reasoning: true,
+		ContextWindow: 272000, MaxOutput: 128000, Reasoning: true,
 		PriceInput: 1.75, PriceOutput: 14.00, PriceCacheRead: 0.175,
 		Speculative: true,
 	},
 	{
 		Provider: "openai", ID: "gpt-5.3", DisplayName: "GPT-5.3",
-		ContextWindow: 400000, MaxOutput: 128000, Reasoning: true,
+		ContextWindow: 272000, MaxOutput: 128000, Reasoning: true,
 		PriceInput: 1.75, PriceOutput: 14.00, PriceCacheRead: 0.175,
 		Speculative: true,
 	},
 	{
 		Provider: "openai", ID: "gpt-5.4", DisplayName: "GPT-5.4",
-		ContextWindow: 400000, MaxOutput: 128000, Reasoning: true,
+		// ContextWindow: 272k across every route we support (OpenAI
+		// direct API and the ChatGPT Codex OAuth backend).
+		ContextWindow: 272000, MaxOutput: 128000, Reasoning: true,
 		PriceInput: 2.50, PriceOutput: 15.00, PriceCacheRead: 0.25,
 		Speculative: true,
 	},
 	{
 		Provider: "openai", ID: "gpt-5.4-mini", DisplayName: "GPT-5.4 mini",
-		ContextWindow: 400000, MaxOutput: 128000, Reasoning: true,
+		// ContextWindow: 400k on the OpenAI direct API, 272k on the
+		// ChatGPT Codex OAuth backend. We pin to the smaller Codex
+		// cap so the context-usage meter is honest under subscription
+		// auth; direct-API users simply see a conservative headroom
+		// estimate rather than an inflated one.
+		ContextWindow: 272000, MaxOutput: 128000, Reasoning: true,
 		PriceInput: 0.75, PriceOutput: 4.50, PriceCacheRead: 0.075,
 		Speculative: true,
 	},