1# zut providers
2
3zut ships with built-in providers and a model catalog. You can select models
4with `/model`, list them with `zut --list-models`, and add private models in
5`$ZUT_HOME/models.json`.
6
7## HTTP proxy
8
9Set one global proxy for zut-managed HTTP and HTTPS traffic with the `http_proxy` key in `$ZUT_HOME/config.json`:
10
11```json
12{
13  "http_proxy": "http://127.0.0.1:7890"
14}
15```
16
17zut applies this setting at startup. Existing `HTTP_PROXY`, `HTTPS_PROXY`, `http_proxy`, and `https_proxy` environment variables take precedence for their corresponding protocol. Standard `NO_PROXY` and `no_proxy` bypass lists remain effective. Restart zut after changing the setting.
18
19`config.json` is not a credential store. If the proxy URL contains a username or password, prefer setting the proxy environment variables in a protected shell or service configuration rather than saving those credentials in `config.json`.
20
21## Login methods
22
23Use `/login` in interactive mode. Type in either provider picker to filter the list by provider ID or display name.
24
25- `api key`: stores an API key in `$ZUT_HOME/auth.json` when the provider uses a normal key.
26- `subscription`: stores OAuth credentials for subscription-backed providers.
27
28Use `/logout` to remove stored credentials.
29
30Some providers need more than a single pasted key. For those providers,
31`/login` shows setup instructions instead of opening a localhost browser form.
32This avoids broken browser flows in SSH, containers, and `kubectl exec`
33sessions.
34
35### Command-backed API keys
36
37A provider can obtain its API key from a password manager or another local program. Configure this directly in `$ZUT_HOME/auth.json`:
38
39```json
40{
41  "openai": {
42    "api_key_command": {
43      "program": "op",
44      "args": ["read", "op://Work/OpenAI/credential"],
45      "timeout_ms": 120000
46    }
47  }
48}
49```
50
51Custom provider credentials use the same object under `additional_api_key_creds`:
52
53```json
54{
55  "additional_api_key_creds": {
56    "my-company": {
57      "api_key_command": {
58        "program": "secret-tool",
59        "args": ["read", "my-company-api-key"]
60      }
61    }
62  }
63}
64```
65
66The program is started directly rather than through a shell. The timeout defaults to 120 seconds. It must write one non-empty line to stdout, with no more than 64 KiB of output. Trailing CR/LF characters are removed. Successful results stay only in process memory and are reused until zut exits.
67
68Command-backed credentials count as logged in without being executed. zut materializes one only when selecting that provider; background model discovery skips it. `/login` with a normal key replaces the command, while `/logout` clears it. Because `auth.json` can cause program execution, keep it user-writable only and do not use files from untrusted sources.
69
70Setup-instruction providers:
71
72- Amazon Bedrock
73- Google Vertex AI
74- Cloudflare Workers AI
75- Cloudflare AI Gateway
76- Azure OpenAI Responses
77
78## Subscription providers
79
80These providers support subscription login:
81
82| Provider | Notes |
83| --- | --- |
84| Anthropic | Claude Pro/Max OAuth credentials. |
85| OpenAI Codex | ChatGPT Plus/Pro Codex subscription route. Separate from the OpenAI API-key provider. |
86| Kimi | Kimi subscription login. |
87| xAI | SuperGrok or X Premium device-code login. The browser URL is prefilled with the device code. |
88| GitHub Copilot | GitHub Copilot token flow. |
89
90OAuth tokens are stored in `$ZUT_HOME/auth.json` and refreshed when refresh is
91available.
92
93## API-key providers
94
95These providers can use environment variables. Simple API-key providers can
96also be configured through `/login`. Providers that require extra cloud setup
97show instructions and should be configured with environment variables.
98
99| Provider | Environment variable | Stored key |
100| --- | --- | --- |
101| Anthropic | `ANTHROPIC_API_KEY` | `anthropic` |
102| OpenAI | `OPENAI_API_KEY` | `openai` |
103| OpenAI Responses | `OPENAI_API_KEY` | `openai-responses` |
104| Kimi | `KIMI_API_KEY` or `MOONSHOT_API_KEY` | `kimi` |
105| Google Gemini | `GEMINI_API_KEY` or `GOOGLE_API_KEY` | `google` |
106| DeepSeek | `DEEPSEEK_API_KEY` | `deepseek` |
107| Moonshot AI | `MOONSHOT_API_KEY` | `moonshotai` |
108| Moonshot AI China | `MOONSHOT_API_KEY` | `moonshotai-cn` |
109| Groq | `GROQ_API_KEY` | `groq` |
110| xAI | `XAI_API_KEY` | `xai` |
111| Cerebras | `CEREBRAS_API_KEY` | `cerebras` |
112| Together AI | `TOGETHER_API_KEY` | `together` |
113| Hugging Face | `HF_TOKEN` | `huggingface` |
114| OpenRouter | `OPENROUTER_API_KEY` | `openrouter` |
115| Mistral | `MISTRAL_API_KEY` | `mistral` |
116| ZAI | `ZAI_API_KEY` | `zai` |
117| Xiaomi MiMo | `XIAOMI_API_KEY` | `xiaomi` |
118| Xiaomi Token Plan Amsterdam | `XIAOMI_TOKEN_PLAN_AMS_API_KEY` | `xiaomi-token-plan-ams` |
119| Xiaomi Token Plan China | `XIAOMI_TOKEN_PLAN_CN_API_KEY` | `xiaomi-token-plan-cn` |
120| Xiaomi Token Plan Singapore | `XIAOMI_TOKEN_PLAN_SGP_API_KEY` | `xiaomi-token-plan-sgp` |
121| MiniMax | `MINIMAX_API_KEY` | `minimax` |
122| MiniMax China | `MINIMAX_CN_API_KEY` or `MINIMAX_API_KEY` | `minimax-cn` |
123| Fireworks | `FIREWORKS_API_KEY` | `fireworks` |
124| Vercel AI Gateway | `AI_GATEWAY_API_KEY` | `vercel-ai-gateway` |
125| OpenCode Zen | `OPENCODE_API_KEY` | `opencode` |
126| OpenCode Go | `OPENCODE_API_KEY` | `opencode-go` |
127| GitHub Copilot token | `COPILOT_GITHUB_TOKEN` or `GITHUB_COPILOT_TOKEN` | `github-copilot` |
128| Cloudflare Workers AI | `CLOUDFLARE_API_KEY` | `cloudflare-workers-ai` |
129| Cloudflare AI Gateway | `CLOUDFLARE_API_KEY` | `cloudflare-ai-gateway` |
130| Azure OpenAI Responses | `AZURE_OPENAI_API_KEY` | `azure-openai-responses` |
131
132Example:
133
134```bash
135export OPENROUTER_API_KEY=...
136zut --provider openrouter
137```
138
139### OpenCode Go model catalog
140
141OpenCode Go is synchronized instead of using a hand-maintained model list. When
142`OPENCODE_API_KEY` or a configured OpenCode Go credential is available, zut
143reads the model IDs published by
144[`https://opencode.ai/zen/go/v1/models`](https://opencode.ai/zen/go/v1/models)
145and enriches them with names, limits, prices, reasoning options, and adapter
146metadata from the [Models.dev JSON API](https://models.dev/api.json), which
147powers the [OpenCode Go provider page](https://models.dev/providers/opencode-go/).
148The published list is authoritative for the catalog and picker, but it is not
149an entitlement check: consent, regional eligibility, quota, and account policy
150can still reject an inference request.
151
152Models.dev adapter metadata selects the wire protocol for each model:
153`@ai-sdk/openai` uses OpenAI Responses, `@ai-sdk/openai-compatible` uses Chat
154Completions, and `@ai-sdk/anthropic` uses Anthropic Messages. If that metadata
155is unavailable, zut keeps the published model ID and uses only its narrow
156bootstrap fallback; protocol compatibility cannot be guaranteed until the
157metadata is available. An explicit unsupported adapter fails with an
158unsupported-protocol error instead of silently choosing Chat Completions.
159Explicit and discovered base URLs remain the credential-scoped gateway
160endpoint; public metadata does not redirect requests to another host.
161
162The merged catalog is cached in `$ZUT_HOME/models-cache.json` for 24 hours;
163OpenCode Go entries are scoped to both the credential fingerprint and endpoint.
164`--list-models` waits for a refresh when the cache is stale, while SDK runtimes
165load matching cached metadata before resolving. Synchronous refreshes resolve
166configured `api_key_command` credentials; unsolicited startup refreshes leave
167those commands untouched.
168
169## Fast mode
170
171Use `/fast` or `/settings` to enable **fast mode** for subsequent model calls.
172The setting is off by default and is stored as `fast_mode` in
173`$ZUT_HOME/config.json`.
174
175Fast mode currently requests OpenAI's Fast service tier for OpenAI Chat,
176OpenAI Responses, and OpenAI Codex requests. Enabling it while using another
177provider currently returns an error instead of silently changing that provider's
178request. Subagent children inherit the setting from their parent unless their
179profile sets `fastMode` explicitly.
180
181## GPT-6 Astra
182
183`gpt-6-astra` is available through `openai` (Responses API with an API key)
184and `openai-codex` (ChatGPT subscription). Both catalog entries deliberately
185use a **500,000-token working context budget**, rather than the larger advertised
186maximum, and retain a 128,000-token maximum output. Native reasoning efforts
187`low`, `medium`, `high`, `xhigh`, and `max` are supported. Selecting reasoning off
188omits the effort parameter; it does not disable Astra's reasoning.
189
190Standard API rates per million tokens are $10 input, $50 output, $1 cache reads,
191and $12.50 cache writes. Above **272,000 total input tokens**, including cache
192reads and writes, rates for the entire request become $20 input, $75 output,
193$2 cache reads, and $25 cache writes. The 500k working budget does not avoid
194this pricing tier. Codex catalog dollar estimates use these API-equivalent
195rates, not subscription billing or quota measurements.
196
197Sources: [Astra model specifications](https://developers.openai.com/api/docs/models/gpt-6-astra)
198and [prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching).
199
200## Session identity and caching
201
202zut supplies every provider request with a root cache identity, a
203conversation-thread identity, and a per-prompt turn ID. They are opaque local
204correlations; adapters translate them only when their exact provider, model,
205and endpoint declaration supports it. Resident children share the root cache
206identity but retain distinct thread identities. The OpenAI/Codex Responses
207route uses the root identity as `prompt_cache_key`; declared Codex routes also
208receive their documented session and thread routing headers. OpenCode Go
209requests identify zut with `x-opencode-client` and `User-Agent`, use the stable
210thread (or root cache) ID as `x-opencode-session`, and use the turn ID as
211`x-opencode-request`. A generic OpenAI-compatible endpoint never receives
212these extensions merely because it accepts an OpenAI-shaped request. JSON
213event mode also reports sanitized cache diagnostics (`eligible`, `mode`,
214`transport`, and `continuation`). `eligible`
215is a conservative local prompt-length estimate; OpenAI performs the authoritative
216tokenization and cache decision. These records never include prompts, durable
217IDs, request bodies, or credentials.
218
219When a provider reports cache detail, zut displays cache-read hit rate as
220`cache reads / cache-reporting prompt input`. A shown `0%` is a confirmed
221cache miss; no percentage means the provider or historical session did not
222report cache detail. Cache writes remain separate from that hit rate. Provider
223retention and eviction, request routing, and OpenAI's 1,024-token eligibility
224minimum can still produce a miss even when the stable prefix is unchanged.
225
226On the official public OpenAI Responses endpoint, GPT-5.6 and GPT-6 Astra
227requests include an explicit cache breakpoint after the stable instruction/tool
228prefix while retaining implicit caching for conversation history. The Codex
229subscription route and custom Responses endpoints do not receive these explicit
230cache controls. Changing request-level reasoning effort can still disrupt prefix
231reuse: zut does not yet send cache-preserving `configuration_update` items.
232
233### OpenAI Responses WebSocket mode
234
235For the exact public endpoint `https://api.openai.com/v1/responses`, zut uses
236the Responses API WebSocket mode for the OpenAI Responses route. It keeps one
237connection per conversation thread, sends one `response.create` at a time,
238and continues the most recent completed response with `previous_response_id`
239only when the complete response-context configuration and prior input prefix
240match exactly, including the completed assistant output that precedes a new
241suffix. Compatible continuations send only new tool or user input.
242The request uses `store:false`; its only provider-specific handshake header is
243`Authorization: Bearer ...`, never the
244ChatGPT/Codex account, routing, or session headers.
245
246The capability is deliberately not inherited by `--base-url`, compatible
247providers, Azure, or the ChatGPT Codex subscription endpoint. Those clients
248continue to use their declared HTTP/SSE contracts. A request without a logical
249session ID, a failed WebSocket setup,
250an injected nonstandard RoundTripper, or a configured proxy falls back to
251HTTP/SSE with the same logical session/cache identity. Cancellation closes
252only the active session socket; a normally completed turn keeps it warm for
253the next continuation.
254
255### Responses explicit continuation
256
257When a Responses completion carries an explicit `end_turn:false`, zut treats the finished inference as a continuation (`turn_end` with `stop:"continue"`) and issues another model call in the same turn, reusing its request context, guards, usage accounting, and queued-input boundary. Tool calls still take precedence over the signal, and errors or cancellation never become continuations. A missing, null, or true `end_turn` keeps the existing behavior and ends the turn.
258
259### Responses reasoning replay
260
261You can switch models or providers within the same session. Assistant text and tool exchanges remain available; encrypted reasoning and thought signatures from a different provider or model are omitted from outgoing requests without editing session history. Switching back allows the matching model to replay its own metadata.
262
263Responses requests run with `store:false`, so only reasoning blocks with an encrypted payload and a valid Responses item ID (when present) are replayed. Session loading recovers message provenance from the provider/model metadata recorded alongside turns. Older compacted messages without provenance retain best-effort replay; malformed reasoning IDs are omitted.
264
265## Local llama.cpp router
266
267The `llama.cpp` provider connects to a multi-model router and is separate from the `ollama` provider. Ollama normally uses `http://localhost:11434`; entering that URL for llama.cpp management produces a 404 because Ollama does not implement the router endpoints.
268
269Install a current llama.cpp build. On macOS:
270
271```bash
272brew install llama.cpp
273# use `brew upgrade llama.cpp` for an existing installation
274```
275
276Start `llama-server` without `--model`, `-m`, or `-hf`. Those flags select one model and disable the router behavior zut expects.
277
278```bash
279mkdir -p ~/llama-models
280
281llama-server \
282  --models-dir ~/llama-models \
283  --no-models-autoload \
284  --jinja \
285  --host 127.0.0.1 \
286  --port 8080 \
287  -ngl 999 \
288  -c 32768
289```
290
291This configuration discovers GGUF files below `~/llama-models`, leaves model loading under explicit user control, enables chat templates, requests GPU offload, and sets a 32K context. Tune the GPU layers and context size for the available memory.
292
293Check the management API before opening zut:
294
295```bash
296curl http://127.0.0.1:8080/health
297curl http://127.0.0.1:8080/models
298```
299
300`/models` must return JSON with a `data` array. A 404 indicates the wrong URL, an older server build, or single-model mode.
301
302To save the connection, run `/login`, choose `api key`, and select `llama.cpp`. Enter `http://127.0.0.1:8080`, without `/v1`, followed by an optional bearer key. zut validates the URL and stores both values in `$ZUT_HOME/auth.json`. Environment variables override the saved connection:
303
304```bash
305export LLAMA_BASE_URL=http://127.0.0.1:8080
306export LLAMA_API_KEY=optional-secret
307```
308
309If a key is configured, start the router with the same `--api-key` value. A server bound only to `127.0.0.1` normally does not need authentication.
310
311After login, `/llama` becomes visible. It shows model state, searches Hugging Face for GGUF repositories, offers available quantizations, reports download and load progress, and explicitly loads or unloads models. Press `d` on a router-downloaded cache model to ask the router to remove it after confirmation. Models discovered through `--models-dir` or a preset are not removable through the router API and must be deleted from their configured source. Some llama.cpp versions retain shared Hugging Face repository artifacts such as `mmproj` files after removing the selected GGUF; remove that repository's cache directory manually if those artifacts are no longer needed. `HF_TOKEN` raises Hugging Face API limits and may be required for repository metadata. For gated files, approve access first and provide the authorized token to the `llama-server` process because the router performs the download.
312
313Opening `/model` refreshes the router and displays its loaded models under provider `llama.cpp`. Unloaded models cannot handle inference and therefore remain exclusive to `/llama` until loaded. Inference is sent to the router's derived `/v1` endpoint.
314
315Ollama downloads are stored in Ollama's internal layout and do not automatically become llama.cpp models. Obtain a GGUF copy through `/llama` or place GGUF files under `~/llama-models`, then restart the router after adding files manually.
316
317## Cloud providers
318
319### Amazon Bedrock
320
321Bedrock is configured with AWS credentials, not a generic zut API-key entry.
322Use one of these credential sources:
323
324```bash
325# AWS profile
326export AWS_PROFILE=your-profile
327
328# IAM access keys
329export AWS_ACCESS_KEY_ID=AKIA...
330export AWS_SECRET_ACCESS_KEY=...
331export AWS_SESSION_TOKEN=... # only for temporary credentials
332
333# Bedrock API key bearer token
334export AWS_BEARER_TOKEN_BEDROCK=bedrock-api-key-...
335
336# Region
337export AWS_REGION=us-east-1
338```
339
340ECS task roles, IRSA, and other AWS SDK credential-chain sources are also
341supported.
342
343Example:
344
345```bash
346AWS_BEARER_TOKEN_BEDROCK=bedrock-api-key-... AWS_REGION=us-east-1 \
347  zut --provider amazon-bedrock --model anthropic.claude-sonnet-4-5-20250929-v1:0
348```
349
350Some Bedrock models require regional inference-profile IDs for on-demand
351throughput, such as `us.` or `eu.` prefixed model IDs. zut rewrites known
352families automatically where possible. Claude Opus 5 uses
353`global.anthropic.claude-opus-5`; zut maps the bare foundation-model ID to that
354global profile and supports its adaptive reasoning levels. Explicit profile IDs
355and ARNs are left unchanged.
356
357### Google Vertex AI
358
359Vertex can use a Google API key when available:
360
361```bash
362export GOOGLE_CLOUD_API_KEY=...
363zut --provider google-vertex
364```
365
366For service-account or application-default credentials, set the standard
367Google environment variables used by your deployment.
368
369### Cloudflare AI Gateway
370
371Cloudflare AI Gateway needs a Cloudflare token plus account and gateway IDs:
372
373```bash
374export CLOUDFLARE_API_KEY=...
375export CLOUDFLARE_ACCOUNT_ID=...
376export CLOUDFLARE_GATEWAY_ID=...
377zut --provider cloudflare-ai-gateway
378```
379
380### Cloudflare Workers AI
381
382Workers AI needs a Cloudflare token and account ID:
383
384```bash
385export CLOUDFLARE_API_KEY=...
386export CLOUDFLARE_ACCOUNT_ID=...
387zut --provider cloudflare-workers-ai
388```
389
390### Azure OpenAI Responses
391
392```bash
393export AZURE_OPENAI_API_KEY=...
394export AZURE_OPENAI_BASE_URL=https://your-resource.openai.azure.com
395export AZURE_OPENAI_API_VERSION=v1 # optional, v1 is the default
396zut --provider azure-openai-responses
397```
398
399The provider uses Azure's Responses API. If deployment names differ from zut
400model IDs, map them without changing the catalog:
401
402```bash
403export AZURE_OPENAI_DEPLOYMENT_NAME_MAP='gpt-5.6-luna=luna-preview,gpt-5.6-sol=sol-preview'
404```
405
406## Auth file
407
408Credentials are stored in `$ZUT_HOME/auth.json` with user-only permissions
409when zut creates the file.
410
411Example:
412
413```json
414{
415  "anthropic": { "api_key": "sk-ant-..." },
416  "openai": { "api_key": "sk-..." },
417  "google": { "api_key": "..." },
418  "additional_api_key_creds": {
419    "openrouter": { "api_key": "..." },
420    "mistral": { "api_key": "..." }
421  }
422}
423```
424
425The top-level keys are used for providers with dedicated credential fields.
426Other API-key providers are stored under `additional_api_key_creds`. Prefer
427`/login` so zut writes the correct schema.
428
429## Custom providers and models
430
431Use `$ZUT_HOME/models.json` for private models, deployment aliases, local
432servers, or OpenAI-compatible gateways that are not in the built-in catalog.
433User entries override built-in entries with the same provider and model ID, and
434adding a `models.json` no longer hides the built-in catalog: your entries are
435merged on top of the baked-in and live-discovered models.
436
437A top-level provider key that is not a built-in id defines a custom provider.
438Give it a provider-level `baseUrl` and an `api` wire format (`openai` for
439OpenAI-compatible Chat Completions, the default, `openai-responses` for the
440OpenAI Responses API, or `anthropic` for the Anthropic Messages API). A
441model-level `baseUrl` overrides the provider-level one for that model, and an
442unknown `api` value falls back to `openai` with a warning.
443
444```json
445{
446  "providers": {
447    "my-company": {
448      "baseUrl": "https://llm.mycompany.com/v1",
449      "api": "openai",
450      "models": [
451        { "id": "company-llm-v2", "name": "Company LLM v2" }
452      ]
453    }
454  }
455}
456```
457
458Custom providers are first-class: they appear in `--list-models`, `/model`, and
459`/login`. `models.json` never stores secrets. Supply the key through `/login`,
460`--api-key`, or a derived environment variable in upper snake case, so
461`my-company` reads `MY_COMPANY_API_KEY`. Because many self-hosted gateways do
462not expose a model-list endpoint, custom provider keys are accepted and stored
463without a verification probe; an invalid key surfaces on the first model call.
464
465To retrieve this custom provider's key from a password manager, add a matching
466entry to `$ZUT_HOME/auth.json`:
467
468```json
469{
470  "additional_api_key_creds": {
471    "my-company": {
472      "api_key_command": {
473        "program": "op",
474        "args": ["read", "op://Work/OpenAI/credential"],
475        "timeout_ms": 120000
476      }
477    }
478  }
479}
480```
481
482The provider IDs in `models.json` and `auth.json` must match. Then select the
483custom model directly:
484
485```sh
486zut --provider my-company --model company-llm-v2
487```
488
489## Credential resolution
490
491For each request, zut checks credentials in this order:
492
4931. Explicit CLI key, such as `--api-key`.
4942. Provider-specific environment variables (including derived custom-provider
495   variables such as `MY_COMPANY_API_KEY`).
4963. `$ZUT_HOME/auth.json`, including custom provider keys saved by `/login`.
497
498`models.json` itself never stores credentials; it only describes models and
499their endpoints.
500
501Bedrock then uses the AWS SDK credential chain for the actual request.