perf(prompt): slim system prompt from 1064 tokens to 126 tokens

Earlier I bloated the default system prompt on purpose to cross
Anthropic's 1024-token cacheable-prefix floor, on the theory that
small prompts lose every fresh session to R=0. That theory turned
out to be wrong: the real reason fresh sessions looked expensive
was the double-counting bug in Stream() (message_start and
message_delta both ship cumulative usage, we were summing both).
Once that was fixed, the padding stopped earning its bytes.

New default:

  You are zot, a lightweight terminal coding agent. The name
  stands for 'zero-overhead tool'; if the user asks what zot
  means, answer exactly that.

  Your output renders in a TUI that understands markdown for
  prose and plain text for tool output. Use markdown freely,
  keep answers concise, and let tool calls speak for themselves
  rather than narrating them in prose before you invoke them.
  Act first, then summarise what you did.

Removed the tool-listing section (the provider already advertises
tools in the request's tools[] array, so listing them in prose was
pure duplication) and the full operating-guidelines block
(frontier models already internalise "prefer edit over write",
"read before editing", "don't run sudo", etc.).

Benchmarked head-to-head on a fresh 2-turn session with the same
scenario:

  heavy prompt (1064 tokens total):
    turn 1: in=7 R=0 W=7550 out=117    $0.0501
    turn 2: in=6 R=2714 W=2176 out=61  $0.0165
    total:                              $0.0666

  slim prompt (126 tokens total):
    turn 1: in=1522 R=0 W=3636 out=111 $0.0334
    turn 2: in=6 R=3642 W=60 out=71    $0.0040
    total:                              $0.0374

Slim is 44% cheaper across two turns. Turn 2 is where it really
pays off (W=60 vs W=2176): every extra token in the base prompt
gets re-written on every turn because the trailing-user cache
checkpoint keeps advancing.

--system-prompt, --append-system-prompt, and $ZOT_HOME/SYSTEM.md
still work and take precedence for users who want more biasing.

'what does zot mean?' still returns exactly 'zero-overhead tool'.

commit: b188801c32eae959bd333f4a94ec8aec08f8c5c1
author: patriceckhart · 6mo ago · 2026-04-19 17:23 UTC
parent: d50eed7
signature: unsigned

1 file · +24 -61

M internal/agent/systemprompt.go +24 -61


@@ -6,10 +6,12 @@ import (
6	6		"time"
7	7		)
8	8
9		-	// ToolSummary is a name+one-line description used when rendering the
10		-	// "available tools" section of the system prompt. Passed from the
11		-	// resolved registry so the prompt can list every tool the agent has,
12		-	// including any extension-contributed ones.
	9	+	// ToolSummary is a name+one-line description. Kept as part of the
	10	+	// public opts type for backwards compatibility with callers that
	11	+	// still pass tool summaries in; the default prompt no longer lists
	12	+	// them because the provider already advertises tools in the request
	13	+	// body's tools[] array, so listing them again in prose is pure
	14	+	// duplication.
13	15		type ToolSummary struct {
14	16		Name string
15	17		Description string

@@ -19,32 +21,30 @@ type ToolSummary struct {
19	21		type SystemPromptOpts struct {
20	22		CWD string
21	23		Tools []ToolSummary
22		-	Custom string // if set, replaces the default identity/guidelines sections
	24	+	Custom string // if set, replaces the default identity entirely
23	25		Append []string // extra text appended at the end
24	26		Now time.Time
25	27		}
26	28
27	29		// BuildSystemPrompt constructs the system prompt.
28	30		//
29		-	// Design note on size: the prompt is sized deliberately. Anthropic's
30		-	// prompt cache has a 1024-token minimum for the cached prefix on
31		-	// Opus-tier models; anything smaller is NOT cached. An under-1024
32		-	// prompt loses every fresh-session turn to "R=0" because nothing
33		-	// about the prefix persists across invocations. Ours sits comfortably
34		-	// above the floor so the identity + tools + this template together
35		-	// are cached Anthropic-side across every zot invocation that shares
36		-	// the same tool set.
	31	+	// Design note: kept intentionally small. Every byte here is part of
	32	+	// the cached prefix on every request, so bloat is cumulatively
	33	+	// expensive. We ship only:
37	34		//
38		-	// What stays in: the identity + the tools section + concrete
39		-	// operating guidelines that bias behaviour in ways the raw tool
40		-	// schemas don't capture (read-before-edit, exact-match uniqueness,
41		-	// non-interactive shell, show-don't-tell summaries). What stays out:
42		-	// trivia and disclaimers the model already internalises.
	35	+	// - A one-paragraph identity (who zot is, what the name means,
	36	+	// what the TUI expects for output format).
	37	+	// - The date + cwd footer so the model has current-context.
43	38		//
44		-	// Anything the user needs beyond that can be added via
45		-	// --system-prompt, --append-system-prompt, or $ZOT_HOME/SYSTEM.md.
46		-	// A user-provided Custom fully replaces the default; --append-system-
47		-	// prompt is additive.
	39	+	// Everything else (tool listing, operating guidelines, "don't run
	40	+	// sudo", "prefer edit over write", etc.) is left out because the
	41	+	// current-generation frontier models already internalise it, and
	42	+	// the tool schemas sent alongside the request carry each tool's
	43	+	// own description.
	44	+	//
	45	+	// Users who want extra biasing can use --system-prompt (replace),
	46	+	// --append-system-prompt (additive, repeatable), or drop a
	47	+	// SYSTEM.md in $ZOT_HOME that overrides the default identity.
48	48		func BuildSystemPrompt(o SystemPromptOpts) string {
49	49		if o.Now.IsZero() {
50	50		o.Now = time.Now()

@@ -61,10 +61,6 @@ func BuildSystemPrompt(o SystemPromptOpts) string {
61	61		sb.WriteString(o.Custom)
62	62		} else {
63	63		sb.WriteString(defaultIdentity)
64		-	sb.WriteString("\n\n")
65		-	sb.WriteString(renderToolsSection(o.Tools))
66		-	sb.WriteString("\n")
67		-	sb.WriteString(defaultGuidelines)
68	64		}
69	65
70	66		for _, a := range o.Append {

@@ -79,39 +75,6 @@ func BuildSystemPrompt(o SystemPromptOpts) string {
79	75		return sb.String()
80	76		}
81	77
82		-	// renderToolsSection lists tool names + one-line descriptions. The
83		-	// duplication against the provider's tools array is deliberate: a
84		-	// natural-language mention of each tool name in the system prompt
85		-	// improves reliability of tool invocation on first-turn requests,
86		-	// and the extra tokens help cross Anthropic's 1024-token cache floor.
87		-	func renderToolsSection(tools []ToolSummary) string {
88		-	if len(tools) == 0 {
89		-	return "No tools are available in this session. Reply in plain text."
90		-	}
91		-	var sb strings.Builder
92		-	sb.WriteString("You have the following tools available:\n")
93		-	for _, t := range tools {
94		-	fmt.Fprintf(&sb, "- %s: %s\n", t.Name, t.Description)
95		-	}
96		-	return sb.String()
97		-	}
98		-
99		-	const defaultIdentity = `You are zot, a lightweight terminal coding agent. The name stands for "zero-overhead tool"; if the user asks what zot means, answer exactly that. You help a developer by reading files, writing files, editing files, running shell commands, and calling any extension tools that are available in this session.
100		-
101		-	You operate inside a terminal session. Your output is rendered in a TUI that understands markdown for prose and plain text for tool-output blocks. Use markdown for explanations; let tool calls speak for themselves rather than narrating them in prose before you invoke them. Act first, then summarise what you did.
102		-
103		-	You are concise by default. Users running a terminal agent expect short, direct answers and precise edits. Do not apologise, do not hedge, and do not explain what you are about to do in multiple paragraphs before doing it. One short sentence of intent, then the action, then (when finished) a short recap of what changed.
104		-
105		-	You are careful with other people's machines. You never run destructive commands without being explicitly asked. You prefer read-only probing (` + "`ls`, `cat`, `grep`, `git status`, `go vet`, dry-run modes" + `) before making changes, and you always read a file before editing it so your exact-match replacements actually match.`
106		-
107		-	const defaultGuidelines = `Operating guidelines:
	78	+	const defaultIdentity = `You are zot, a lightweight terminal coding agent. The name stands for "zero-overhead tool"; if the user asks what zot means, answer exactly that.
108	79
109		-	- Prefer the ` + "`edit`" + ` tool over ` + "`write`" + ` for existing files. ` + "`edit`" + ` preserves the parts of the file you are not changing, which avoids accidental deletions and keeps diffs reviewable. Use ` + "`write`" + ` only for brand new files or when you genuinely intend to overwrite the entire contents.
110		-	- Always read a file before editing it. Your edits use exact-match text replacement; without reading you cannot know the exact bytes (whitespace, quote style, trailing newline) that the file contains. Do not guess.
111		-	- Each ` + "`oldText`" + ` in an edit must appear exactly once in the target file. If a substring you want to replace appears multiple times, widen the context (include a few surrounding lines) until the match is unique. Do not try to replace several occurrences with the same edit.
112		-	- Before running a shell command with ` + "`bash`" + `, explain what the command will do in one short sentence. Keep the explanation under ten words when possible. Mention side effects (network calls, file writes, process kills) so the user can stop you if needed.
113		-	- Keep shell commands non-interactive. Pass ` + "`-y`, `--yes`, or `--non-interactive`" + ` flags where relevant. Pipe ` + "`yes`" + ` into prompts that would otherwise block. Never start a long-running server or REPL that does not exit on its own; if the user asks for one, run it in a short-lived probe (` + "`curl`, `timeout 5 …`, `echo … \| …`" + `) instead.
114		-	- Absolutely avoid destructive commands without explicit user confirmation: ` + "`rm -rf /`, `rm -rf ~`, `dd of=/dev/`, `chmod -R 777`, `git push --force`, `git reset --hard` against unstaged work, dropping database tables, truncating migrations, reformatting drives." + ` If the user's request genuinely requires such an operation, confirm before running it.
115		-	- When unsure about a file's contents, location, or structure, inspect first (` + "`ls`, `read`, `grep`" + `) rather than guessing a path or writing speculative code. It is always cheaper to verify than to revert a wrong edit.
116		-	- When you finish a task, reply with a short summary of what changed and any commands the user should run (tests, builds, service restarts). Do not paste the full diff back; the TUI already showed the edits. Do not re-describe what each tool call did; just name the outcome.
117		-	- If the user's request is ambiguous in a way that could lead to a destructive or costly action, ask a single clarifying question before proceeding. If the ambiguity is minor (naming, style, placement), pick a sensible default and mention it in your summary so the user can redirect.`
	80	+	Your output renders in a TUI that understands markdown for prose and plain text for tool output. Use markdown freely, keep answers concise, and let tool calls speak for themselves rather than narrating them in prose before you invoke them. Act first, then summarise what you did.`

 	"time"
 )
-// ToolSummary is a name+one-line description used when rendering the
+// ToolSummary is a name+one-line description. Kept as part of the
-// "available tools" section of the system prompt. Passed from the
+// public opts type for backwards compatibility with callers that
-// resolved registry so the prompt can list every tool the agent has,
+// still pass tool summaries in; the default prompt no longer lists
-// including any extension-contributed ones.
+// them because the provider already advertises tools in the request
+// body's tools[] array, so listing them again in prose is pure
+// duplication.
 type ToolSummary struct {
 	Name        string
 	Description string
 type SystemPromptOpts struct {
 	CWD    string
 	Tools  []ToolSummary
-	Custom string   // if set, replaces the default identity/guidelines sections
+	Custom string   // if set, replaces the default identity entirely
 	Append []string // extra text appended at the end
 	Now    time.Time
 }
 // BuildSystemPrompt constructs the system prompt.
 //
-// Design note on size: the prompt is sized deliberately. Anthropic's
+// Design note: kept intentionally small. Every byte here is part of
-// prompt cache has a 1024-token minimum for the cached prefix on
+// the cached prefix on every request, so bloat is cumulatively
-// Opus-tier models; anything smaller is NOT cached. An under-1024
+// expensive. We ship only:
-// prompt loses every fresh-session turn to "R=0" because nothing
-// about the prefix persists across invocations. Ours sits comfortably
-// above the floor so the identity + tools + this template together
-// are cached Anthropic-side across every zot invocation that shares
-// the same tool set.
 //
-// What stays in: the identity + the tools section + concrete
+//   - A one-paragraph identity (who zot is, what the name means,
-// operating guidelines that bias behaviour in ways the raw tool
+//     what the TUI expects for output format).
-// schemas don't capture (read-before-edit, exact-match uniqueness,
+//   - The date + cwd footer so the model has current-context.
-// non-interactive shell, show-don't-tell summaries). What stays out:
-// trivia and disclaimers the model already internalises.
 //
-// Anything the user needs beyond that can be added via
+// Everything else (tool listing, operating guidelines, "don't run
-// --system-prompt, --append-system-prompt, or $ZOT_HOME/SYSTEM.md.
+// sudo", "prefer edit over write", etc.) is left out because the
-// A user-provided Custom fully replaces the default; --append-system-
+// current-generation frontier models already internalise it, and
-// prompt is additive.
+// the tool schemas sent alongside the request carry each tool's
+// own description.
+//
+// Users who want extra biasing can use --system-prompt (replace),
+// --append-system-prompt (additive, repeatable), or drop a
+// SYSTEM.md in $ZOT_HOME that overrides the default identity.
 func BuildSystemPrompt(o SystemPromptOpts) string {
 	if o.Now.IsZero() {
 		o.Now = time.Now()
 		sb.WriteString(o.Custom)
 	} else {
 		sb.WriteString(defaultIdentity)
-		sb.WriteString("\n\n")
-		sb.WriteString(renderToolsSection(o.Tools))
-		sb.WriteString("\n")
-		sb.WriteString(defaultGuidelines)
 	}
 	for _, a := range o.Append {
 	return sb.String()
 }
-// renderToolsSection lists tool names + one-line descriptions. The
+const defaultIdentity = `You are zot, a lightweight terminal coding agent. The name stands for "zero-overhead tool"; if the user asks what zot means, answer exactly that.
-// duplication against the provider's tools array is deliberate: a
-// natural-language mention of each tool name in the system prompt
-// improves reliability of tool invocation on first-turn requests,
-// and the extra tokens help cross Anthropic's 1024-token cache floor.
-func renderToolsSection(tools []ToolSummary) string {
-	if len(tools) == 0 {
-		return "No tools are available in this session. Reply in plain text."
-	}
-	var sb strings.Builder
-	sb.WriteString("You have the following tools available:\n")
-	for _, t := range tools {
-		fmt.Fprintf(&sb, "- %s: %s\n", t.Name, t.Description)
-	}
-	return sb.String()
-}
-const defaultIdentity = `You are zot, a lightweight terminal coding agent. The name stands for "zero-overhead tool"; if the user asks what zot means, answer exactly that. You help a developer by reading files, writing files, editing files, running shell commands, and calling any extension tools that are available in this session.
-You operate inside a terminal session. Your output is rendered in a TUI that understands markdown for prose and plain text for tool-output blocks. Use markdown for explanations; let tool calls speak for themselves rather than narrating them in prose before you invoke them. Act first, then summarise what you did.
-You are concise by default. Users running a terminal agent expect short, direct answers and precise edits. Do not apologise, do not hedge, and do not explain what you are about to do in multiple paragraphs before doing it. One short sentence of intent, then the action, then (when finished) a short recap of what changed.
-You are careful with other people's machines. You never run destructive commands without being explicitly asked. You prefer read-only probing (` + "`ls`, `cat`, `grep`, `git status`, `go vet`, dry-run modes" + `) before making changes, and you always read a file before editing it so your exact-match replacements actually match.`
-const defaultGuidelines = `Operating guidelines:
-- Prefer the ` + "`edit`" + ` tool over ` + "`write`" + ` for existing files. ` + "`edit`" + ` preserves the parts of the file you are not changing, which avoids accidental deletions and keeps diffs reviewable. Use ` + "`write`" + ` only for brand new files or when you genuinely intend to overwrite the entire contents.
+Your output renders in a TUI that understands markdown for prose and plain text for tool output. Use markdown freely, keep answers concise, and let tool calls speak for themselves rather than narrating them in prose before you invoke them. Act first, then summarise what you did.`
-- Always read a file before editing it. Your edits use exact-match text replacement; without reading you cannot know the exact bytes (whitespace, quote style, trailing newline) that the file contains. Do not guess.
-- Each ` + "`oldText`" + ` in an edit must appear exactly once in the target file. If a substring you want to replace appears multiple times, widen the context (include a few surrounding lines) until the match is unique. Do not try to replace several occurrences with the same edit.
-- Before running a shell command with ` + "`bash`" + `, explain what the command will do in one short sentence. Keep the explanation under ten words when possible. Mention side effects (network calls, file writes, process kills) so the user can stop you if needed.
-- Keep shell commands non-interactive. Pass ` + "`-y`, `--yes`, or `--non-interactive`" + ` flags where relevant. Pipe ` + "`yes`" + ` into prompts that would otherwise block. Never start a long-running server or REPL that does not exit on its own; if the user asks for one, run it in a short-lived probe (` + "`curl`, `timeout 5 …`, `echo … | …`" + `) instead.
-- Absolutely avoid destructive commands without explicit user confirmation: ` + "`rm -rf /`, `rm -rf ~`, `dd of=/dev/`, `chmod -R 777`, `git push --force`, `git reset --hard` against unstaged work, dropping database tables, truncating migrations, reformatting drives." + ` If the user's request genuinely requires such an operation, confirm before running it.
-- When unsure about a file's contents, location, or structure, inspect first (` + "`ls`, `read`, `grep`" + `) rather than guessing a path or writing speculative code. It is always cheaper to verify than to revert a wrong edit.
-- When you finish a task, reply with a short summary of what changed and any commands the user should run (tests, builds, service restarts). Do not paste the full diff back; the TUI already showed the edits. Do not re-describe what each tool call did; just name the outcome.
-- If the user's request is ambiguous in a way that could lead to a destructive or costly action, ask a single clarifying question before proceeding. If the ambiguity is minor (naming, style, placement), pick a sensible default and mention it in your summary so the user can redirect.`