1# zut providers
2
3zut ships with built-in providers and a model catalog. You can select models
4with `/model`, list them with `zut --list-models`, and add private models in
5`$ZUT_HOME/models.json`.
6
7## HTTP proxy
8
9Set one global proxy for zut-managed HTTP and HTTPS traffic with the `http_proxy` key in `$ZUT_HOME/config.json`:
10
11```json
12{
13  "http_proxy": "http://127.0.0.1:7890"
14}
15```
16
17zut applies this setting at startup. Existing `HTTP_PROXY`, `HTTPS_PROXY`, `http_proxy`, and `https_proxy` environment variables take precedence for their corresponding protocol. Standard `NO_PROXY` and `no_proxy` bypass lists remain effective. Restart zut after changing the setting.
18
19`config.json` is not a credential store. If the proxy URL contains a username or password, prefer setting the proxy environment variables in a protected shell or service configuration rather than saving those credentials in `config.json`.
20
21## Login methods
22
23Use `/login` in interactive mode. Type in either provider picker to filter the list by provider ID or display name.
24
25- `api key`: stores an API key in `$ZUT_HOME/auth.json` when the provider uses a normal key.
26- `subscription`: stores OAuth credentials for subscription-backed providers.
27
28Use `/logout` to remove stored credentials.
29
30Some providers need more than a single pasted key. For those providers,
31`/login` shows setup instructions instead of opening a localhost browser form.
32This avoids broken browser flows in SSH, containers, and `kubectl exec`
33sessions.
34
35### Command-backed API keys
36
37A provider can obtain its API key from a password manager or another local program. Configure this directly in `$ZUT_HOME/auth.json`:
38
39```json
40{
41  "openai": {
42    "api_key_command": {
43      "program": "op",
44      "args": ["read", "op://Work/OpenAI/credential"],
45      "timeout_ms": 120000
46    }
47  }
48}
49```
50
51Custom provider credentials use the same object under `additional_api_key_creds`:
52
53```json
54{
55  "additional_api_key_creds": {
56    "my-company": {
57      "api_key_command": {
58        "program": "secret-tool",
59        "args": ["read", "my-company-api-key"]
60      }
61    }
62  }
63}
64```
65
66The program is started directly rather than through a shell. The timeout defaults to 120 seconds. It must write one non-empty line to stdout, with no more than 64 KiB of output. Trailing CR/LF characters are removed. Successful results stay only in process memory and are reused until zut exits.
67
68Command-backed credentials count as logged in without being executed. zut materializes one only when selecting that provider; background model discovery skips it. `/login` with a normal key replaces the command, while `/logout` clears it. Because `auth.json` can cause program execution, keep it user-writable only and do not use files from untrusted sources.
69
70Setup-instruction providers:
71
72- Amazon Bedrock
73- Google Vertex AI
74- Cloudflare Workers AI
75- Cloudflare AI Gateway
76- Azure OpenAI Responses
77
78## Subscription providers
79
80These providers support subscription login:
81
82| Provider | Notes |
83| --- | --- |
84| Anthropic | Claude Pro/Max OAuth credentials. |
85| OpenAI Codex | ChatGPT Plus/Pro Codex subscription route. Separate from the OpenAI API-key provider. |
86| Kimi | Kimi subscription login. |
87| xAI | SuperGrok or X Premium device-code login. The browser URL is prefilled with the device code. |
88| GitHub Copilot | GitHub Copilot token flow. |
89
90OAuth tokens are stored in `$ZUT_HOME/auth.json` and refreshed when refresh is
91available.
92
93## API-key providers
94
95These providers can use environment variables. Simple API-key providers can
96also be configured through `/login`. Providers that require extra cloud setup
97show instructions and should be configured with environment variables.
98
99| Provider | Environment variable | Stored key |
100| --- | --- | --- |
101| Anthropic | `ANTHROPIC_API_KEY` | `anthropic` |
102| OpenAI | `OPENAI_API_KEY` | `openai` |
103| OpenAI Responses | `OPENAI_API_KEY` | `openai-responses` |
104| Kimi | `KIMI_API_KEY` or `MOONSHOT_API_KEY` | `kimi` |
105| Google Gemini | `GEMINI_API_KEY` or `GOOGLE_API_KEY` | `google` |
106| DeepSeek | `DEEPSEEK_API_KEY` | `deepseek` |
107| Ollama Cloud | `OLLAMA_API_KEY` | `ollama` |
108| Moonshot AI | `MOONSHOT_API_KEY` | `moonshotai` |
109| Moonshot AI China | `MOONSHOT_API_KEY` | `moonshotai-cn` |
110| Groq | `GROQ_API_KEY` | `groq` |
111| xAI | `XAI_API_KEY` | `xai` |
112| Cerebras | `CEREBRAS_API_KEY` | `cerebras` |
113| Together AI | `TOGETHER_API_KEY` | `together` |
114| Hugging Face | `HF_TOKEN` | `huggingface` |
115| OpenRouter | `OPENROUTER_API_KEY` | `openrouter` |
116| Mistral | `MISTRAL_API_KEY` | `mistral` |
117| ZAI | `ZAI_API_KEY` | `zai` |
118| Xiaomi MiMo | `XIAOMI_API_KEY` | `xiaomi` |
119| Xiaomi Token Plan Amsterdam | `XIAOMI_TOKEN_PLAN_AMS_API_KEY` | `xiaomi-token-plan-ams` |
120| Xiaomi Token Plan China | `XIAOMI_TOKEN_PLAN_CN_API_KEY` | `xiaomi-token-plan-cn` |
121| Xiaomi Token Plan Singapore | `XIAOMI_TOKEN_PLAN_SGP_API_KEY` | `xiaomi-token-plan-sgp` |
122| MiniMax | `MINIMAX_API_KEY` | `minimax` |
123| MiniMax China | `MINIMAX_CN_API_KEY` or `MINIMAX_API_KEY` | `minimax-cn` |
124| Fireworks | `FIREWORKS_API_KEY` | `fireworks` |
125| Vercel AI Gateway | `AI_GATEWAY_API_KEY` | `vercel-ai-gateway` |
126| OpenCode Zen | `OPENCODE_API_KEY` | `opencode` |
127| OpenCode Go | `OPENCODE_API_KEY` | `opencode-go` |
128| GitHub Copilot token | `COPILOT_GITHUB_TOKEN` or `GITHUB_COPILOT_TOKEN` | `github-copilot` |
129| Cloudflare Workers AI | `CLOUDFLARE_API_KEY` | `cloudflare-workers-ai` |
130| Cloudflare AI Gateway | `CLOUDFLARE_API_KEY` | `cloudflare-ai-gateway` |
131| Azure OpenAI Responses | `AZURE_OPENAI_API_KEY` | `azure-openai-responses` |
132
133Example:
134
135```bash
136export OPENROUTER_API_KEY=...
137zut --provider openrouter
138```
139
140### Ollama local vs Cloud
141
142The `ollama` provider serves both local and Cloud models over the OpenAI-compatible `/v1` API. Secrets are sent only as `Authorization: Bearer` headers, never in URLs or logs.
143
144- No key: `http://localhost:11434`, no auth (local models).
145- Real key (`--api-key`, `OLLAMA_API_KEY`, or `/login` → Ollama) with no explicit endpoint: `https://ollama.com/v1` (Cloud models).
146- `--base-url` always wins; a `baseUrl` in `models.json` (model- or provider-level) wins over the automatic default.
147
148### OpenCode Go model catalog
149
150OpenCode Go is synchronized instead of using a hand-maintained model list. When
151`OPENCODE_API_KEY` or a configured OpenCode Go credential is available, zut
152reads the model IDs published by
153[`https://opencode.ai/zen/go/v1/models`](https://opencode.ai/zen/go/v1/models)
154and enriches them with names, limits, prices, reasoning options, and adapter
155metadata from the [Models.dev JSON API](https://models.dev/api.json), which
156powers the [OpenCode Go provider page](https://models.dev/providers/opencode-go/).
157The published list is authoritative for the catalog and picker, but it is not
158an entitlement check: consent, regional eligibility, quota, and account policy
159can still reject an inference request.
160
161Models.dev adapter metadata selects the wire protocol for each model:
162`@ai-sdk/openai` uses OpenAI Responses, `@ai-sdk/openai-compatible` uses Chat
163Completions, and `@ai-sdk/anthropic` uses Anthropic Messages. If that metadata
164is unavailable, zut keeps the published model ID and uses only its narrow
165bootstrap fallback; protocol compatibility cannot be guaranteed until the
166metadata is available. An explicit unsupported adapter fails with an
167unsupported-protocol error instead of silently choosing Chat Completions.
168Explicit and discovered base URLs remain the credential-scoped gateway
169endpoint; public metadata does not redirect requests to another host.
170
171The merged catalog is cached in `$ZUT_HOME/models-cache.json` for 24 hours;
172OpenCode Go entries are scoped to both the credential fingerprint and endpoint.
173`--list-models` waits for a refresh when the cache is stale, while SDK runtimes
174load matching cached metadata before resolving. Synchronous refreshes resolve
175configured `api_key_command` credentials; unsolicited startup refreshes leave
176those commands untouched.
177
178## Fast mode
179
180Use `/fast` or `/settings` to enable **fast mode** for subsequent model calls.
181The setting is off by default and is stored as `fast_mode` in
182`$ZUT_HOME/config.json`.
183
184Fast mode currently requests OpenAI's Fast service tier for OpenAI Chat,
185OpenAI Responses, and OpenAI Codex requests. Enabling it while using another
186provider currently returns an error instead of silently changing that provider's
187request. Subagent children inherit the setting from their parent unless their
188profile sets `fastMode` explicitly.
189
190## GPT-6 Astra
191
192`gpt-6-astra` is available through `openai` (Responses API with an API key)
193and `openai-codex` (ChatGPT subscription). Both catalog entries deliberately
194use a **500,000-token working context budget**, rather than the larger advertised
195maximum, and retain a 128,000-token maximum output. Native reasoning efforts
196`low`, `medium`, `high`, `xhigh`, and `max` are supported. Selecting reasoning off
197omits the effort parameter; it does not disable Astra's reasoning.
198
199Standard API rates per million tokens are $10 input, $50 output, $1 cache reads,
200and $12.50 cache writes. Above **272,000 total input tokens**, including cache
201reads and writes, rates for the entire request become $20 input, $75 output,
202$2 cache reads, and $25 cache writes. The 500k working budget does not avoid
203this pricing tier. Codex catalog dollar estimates use these API-equivalent
204rates, not subscription billing or quota measurements.
205
206Sources: [Astra model specifications](https://developers.openai.com/api/docs/models/gpt-6-astra)
207and [prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching).
208
209## Session identity and caching
210
211zut supplies every provider request with a root cache identity, a
212conversation-thread identity, and a per-prompt turn ID. They are opaque local
213correlations; adapters translate them only when their exact provider, model,
214and endpoint declaration supports it. Resident children share the root cache
215identity but retain distinct thread identities. The OpenAI/Codex Responses
216route uses the root identity as `prompt_cache_key`; declared Codex routes also
217receive their documented session and thread routing headers. OpenCode Go
218requests identify zut with `x-opencode-client` and `User-Agent`, use the stable
219thread (or root cache) ID as `x-opencode-session`, and use the turn ID as
220`x-opencode-request`. A generic OpenAI-compatible endpoint never receives
221these extensions merely because it accepts an OpenAI-shaped request. JSON
222event mode also reports sanitized cache diagnostics (`eligible`, `mode`,
223`transport`, and `continuation`). `eligible`
224is a conservative local prompt-length estimate; OpenAI performs the authoritative
225tokenization and cache decision. These records never include prompts, durable
226IDs, request bodies, or credentials.
227
228When a provider reports cache detail, zut displays cache-read hit rate as
229`cache reads / cache-reporting prompt input`. A shown `0%` is a confirmed
230cache miss; no percentage means the provider or historical session did not
231report cache detail. Cache writes remain separate from that hit rate. Provider
232retention and eviction, request routing, and OpenAI's 1,024-token eligibility
233minimum can still produce a miss even when the stable prefix is unchanged.
234
235On the official public OpenAI Responses endpoint, GPT-5.6 and GPT-6 Astra
236requests include an explicit cache breakpoint after the stable instruction/tool
237prefix while retaining implicit caching for conversation history. The Codex
238subscription route and custom Responses endpoints do not receive these explicit
239cache controls. Changing request-level reasoning effort can still disrupt prefix
240reuse: zut does not yet send cache-preserving `configuration_update` items.
241
242### OpenAI Responses WebSocket mode
243
244For the exact public endpoint `https://api.openai.com/v1/responses`, zut uses
245the Responses API WebSocket mode for the OpenAI Responses route. It keeps one
246connection per conversation thread, sends one `response.create` at a time,
247and continues the most recent completed response with `previous_response_id`
248only when the complete response-context configuration and prior input prefix
249match exactly, including the completed assistant output that precedes a new
250suffix. Compatible continuations send only new tool or user input.
251The request uses `store:false`; its only provider-specific handshake header is
252`Authorization: Bearer ...`, never the
253ChatGPT/Codex account, routing, or session headers.
254
255The capability is deliberately not inherited by `--base-url`, compatible
256providers, Azure, or the ChatGPT Codex subscription endpoint. Those clients
257continue to use their declared HTTP/SSE contracts. A request without a logical
258session ID, a failed WebSocket setup,
259an injected nonstandard RoundTripper, or a configured proxy falls back to
260HTTP/SSE with the same logical session/cache identity. Cancellation closes
261only the active session socket; a normally completed turn keeps it warm for
262the next continuation.
263
264### Responses explicit continuation
265
266When a Responses completion carries an explicit `end_turn:false`, zut treats the finished inference as a continuation (`turn_end` with `stop:"continue"`) and issues another model call in the same turn, reusing its request context, guards, usage accounting, and queued-input boundary. Tool calls still take precedence over the signal, and errors or cancellation never become continuations. A missing, null, or true `end_turn` keeps the existing behavior and ends the turn.
267
268### Responses reasoning replay
269
270You can switch models or providers within the same session. Assistant text and tool exchanges remain available; encrypted reasoning and thought signatures from a different provider or model are omitted from outgoing requests without editing session history. Switching back allows the matching model to replay its own metadata.
271
272Responses requests run with `store:false`, so only reasoning blocks with an encrypted payload and a valid Responses item ID (when present) are replayed. Session loading recovers message provenance from the provider/model metadata recorded alongside turns. Older compacted messages without provenance retain best-effort replay; malformed reasoning IDs are omitted.
273
274## Local llama.cpp router
275
276The `llama.cpp` provider connects to a multi-model router and is separate from the `ollama` provider. Ollama normally uses `http://localhost:11434`; entering that URL for llama.cpp management produces a 404 because Ollama does not implement the router endpoints.
277
278Install a current llama.cpp build. On macOS:
279
280```bash
281brew install llama.cpp
282# use `brew upgrade llama.cpp` for an existing installation
283```
284
285Start `llama-server` without `--model`, `-m`, or `-hf`. Those flags select one model and disable the router behavior zut expects.
286
287```bash
288mkdir -p ~/llama-models
289
290llama-server \
291  --models-dir ~/llama-models \
292  --no-models-autoload \
293  --jinja \
294  --host 127.0.0.1 \
295  --port 8080 \
296  -ngl 999 \
297  -c 32768
298```
299
300This configuration discovers GGUF files below `~/llama-models`, leaves model loading under explicit user control, enables chat templates, requests GPU offload, and sets a 32K context. Tune the GPU layers and context size for the available memory.
301
302Check the management API before opening zut:
303
304```bash
305curl http://127.0.0.1:8080/health
306curl http://127.0.0.1:8080/models
307```
308
309`/models` must return JSON with a `data` array. A 404 indicates the wrong URL, an older server build, or single-model mode.
310
311To save the connection, run `/login`, choose `api key`, and select `llama.cpp`. Enter `http://127.0.0.1:8080`, without `/v1`, followed by an optional bearer key. zut validates the URL and stores both values in `$ZUT_HOME/auth.json`. Environment variables override the saved connection:
312
313```bash
314export LLAMA_BASE_URL=http://127.0.0.1:8080
315export LLAMA_API_KEY=optional-secret
316```
317
318If a key is configured, start the router with the same `--api-key` value. A server bound only to `127.0.0.1` normally does not need authentication.
319
320After login, `/llama` becomes visible. It shows model state, searches Hugging Face for GGUF repositories, offers available quantizations, reports download and load progress, and explicitly loads or unloads models. Press `d` on a router-downloaded cache model to ask the router to remove it after confirmation. Models discovered through `--models-dir` or a preset are not removable through the router API and must be deleted from their configured source. Some llama.cpp versions retain shared Hugging Face repository artifacts such as `mmproj` files after removing the selected GGUF; remove that repository's cache directory manually if those artifacts are no longer needed. `HF_TOKEN` raises Hugging Face API limits and may be required for repository metadata. For gated files, approve access first and provide the authorized token to the `llama-server` process because the router performs the download.
321
322Opening `/model` refreshes the router and displays its loaded models under provider `llama.cpp`. Unloaded models cannot handle inference and therefore remain exclusive to `/llama` until loaded. Inference is sent to the router's derived `/v1` endpoint.
323
324Ollama downloads are stored in Ollama's internal layout and do not automatically become llama.cpp models. Obtain a GGUF copy through `/llama` or place GGUF files under `~/llama-models`, then restart the router after adding files manually.
325
326## Cloud providers
327
328### Amazon Bedrock
329
330Bedrock is configured with AWS credentials, not a generic zut API-key entry.
331Use one of these credential sources:
332
333```bash
334# AWS profile
335export AWS_PROFILE=your-profile
336
337# IAM access keys
338export AWS_ACCESS_KEY_ID=AKIA...
339export AWS_SECRET_ACCESS_KEY=...
340export AWS_SESSION_TOKEN=... # only for temporary credentials
341
342# Bedrock API key bearer token
343export AWS_BEARER_TOKEN_BEDROCK=bedrock-api-key-...
344
345# Region
346export AWS_REGION=us-east-1
347```
348
349ECS task roles, IRSA, and other AWS SDK credential-chain sources are also
350supported.
351
352Example:
353
354```bash
355AWS_BEARER_TOKEN_BEDROCK=bedrock-api-key-... AWS_REGION=us-east-1 \
356  zut --provider amazon-bedrock --model anthropic.claude-sonnet-4-5-20250929-v1:0
357```
358
359Some Bedrock models require regional inference-profile IDs for on-demand
360throughput, such as `us.` or `eu.` prefixed model IDs. zut rewrites known
361families automatically where possible. Claude Opus 5 uses
362`global.anthropic.claude-opus-5`; zut maps the bare foundation-model ID to that
363global profile and supports its adaptive reasoning levels. Explicit profile IDs
364and ARNs are left unchanged.
365
366### Google Vertex AI
367
368Vertex can use a Google API key when available:
369
370```bash
371export GOOGLE_CLOUD_API_KEY=...
372zut --provider google-vertex
373```
374
375For service-account or application-default credentials, set the standard
376Google environment variables used by your deployment.
377
378### Cloudflare AI Gateway
379
380Cloudflare AI Gateway needs a Cloudflare token plus account and gateway IDs:
381
382```bash
383export CLOUDFLARE_API_KEY=...
384export CLOUDFLARE_ACCOUNT_ID=...
385export CLOUDFLARE_GATEWAY_ID=...
386zut --provider cloudflare-ai-gateway
387```
388
389### Cloudflare Workers AI
390
391Workers AI needs a Cloudflare token and account ID:
392
393```bash
394export CLOUDFLARE_API_KEY=...
395export CLOUDFLARE_ACCOUNT_ID=...
396zut --provider cloudflare-workers-ai
397```
398
399### Azure OpenAI Responses
400
401```bash
402export AZURE_OPENAI_API_KEY=...
403export AZURE_OPENAI_BASE_URL=https://your-resource.openai.azure.com
404export AZURE_OPENAI_API_VERSION=v1 # optional, v1 is the default
405zut --provider azure-openai-responses
406```
407
408The provider uses Azure's Responses API. If deployment names differ from zut
409model IDs, map them without changing the catalog:
410
411```bash
412export AZURE_OPENAI_DEPLOYMENT_NAME_MAP='gpt-5.6-luna=luna-preview,gpt-5.6-sol=sol-preview'
413```
414
415## Auth file
416
417Credentials are stored in `$ZUT_HOME/auth.json` with user-only permissions
418when zut creates the file.
419
420Example:
421
422```json
423{
424  "anthropic": { "api_key": "sk-ant-..." },
425  "openai": { "api_key": "sk-..." },
426  "google": { "api_key": "..." },
427  "additional_api_key_creds": {
428    "openrouter": { "api_key": "..." },
429    "mistral": { "api_key": "..." }
430  }
431}
432```
433
434The top-level keys are used for providers with dedicated credential fields.
435Other API-key providers are stored under `additional_api_key_creds`. Prefer
436`/login` so zut writes the correct schema.
437
438## Custom providers and models
439
440Use `$ZUT_HOME/models.json` for private models, deployment aliases, local
441servers, or OpenAI-compatible gateways that are not in the built-in catalog.
442User entries override built-in entries with the same provider and model ID, and
443adding a `models.json` no longer hides the built-in catalog: your entries are
444merged on top of the baked-in and live-discovered models.
445
446A top-level provider key that is not a built-in id defines a custom provider.
447Give it a provider-level `baseUrl` and an `api` wire format (`openai` for
448OpenAI-compatible Chat Completions, the default, `openai-responses` for the
449OpenAI Responses API, or `anthropic` for the Anthropic Messages API). A
450model-level `baseUrl` overrides the provider-level one for that model, and an
451unknown `api` value falls back to `openai` with a warning.
452
453```json
454{
455  "providers": {
456    "my-company": {
457      "baseUrl": "https://llm.mycompany.com/v1",
458      "api": "openai",
459      "models": [
460        { "id": "company-llm-v2", "name": "Company LLM v2" }
461      ]
462    }
463  }
464}
465```
466
467Custom providers are first-class: they appear in `--list-models`, `/model`, and
468`/login`. `models.json` never stores secrets. Supply the key through `/login`,
469`--api-key`, or a derived environment variable in upper snake case, so
470`my-company` reads `MY_COMPANY_API_KEY`. Because many self-hosted gateways do
471not expose a model-list endpoint, custom provider keys are accepted and stored
472without a verification probe; an invalid key surfaces on the first model call.
473
474To retrieve this custom provider's key from a password manager, add a matching
475entry to `$ZUT_HOME/auth.json`:
476
477```json
478{
479  "additional_api_key_creds": {
480    "my-company": {
481      "api_key_command": {
482        "program": "op",
483        "args": ["read", "op://Work/OpenAI/credential"],
484        "timeout_ms": 120000
485      }
486    }
487  }
488}
489```
490
491The provider IDs in `models.json` and `auth.json` must match. Then select the
492custom model directly:
493
494```sh
495zut --provider my-company --model company-llm-v2
496```
497
498## Credential resolution
499
500For each request, zut checks credentials in this order:
501
5021. Explicit CLI key, such as `--api-key`.
5032. Provider-specific environment variables (including derived custom-provider
504   variables such as `MY_COMPANY_API_KEY`).
5053. `$ZUT_HOME/auth.json`, including custom provider keys saved by `/login`.
506
507`models.json` itself never stores credentials; it only describes models and
508their endpoints.
509
510Bedrock then uses the AWS SDK credential chain for the actual request.