1# zut providers
2
3zut ships with built-in providers and a model catalog. You can select models
4with `/model`, list them with `zut --list-models`, and add private models in
5`$ZUT_HOME/models.json`.
6
7## HTTP proxy
8
9Set one global proxy for zut-managed HTTP and HTTPS traffic with the `http_proxy` key in `$ZUT_HOME/config.json`:
10
11```json
12{
13  "http_proxy": "http://127.0.0.1:7890"
14}
15```
16
17zut applies this setting at startup. Existing `HTTP_PROXY`, `HTTPS_PROXY`, `http_proxy`, and `https_proxy` environment variables take precedence for their corresponding protocol. Standard `NO_PROXY` and `no_proxy` bypass lists remain effective. Restart zut after changing the setting.
18
19`config.json` is not a credential store. If the proxy URL contains a username or password, prefer setting the proxy environment variables in a protected shell or service configuration rather than saving those credentials in `config.json`.
20
21## Login methods
22
23Use `/login` in interactive mode. Type in either provider picker to filter the list by provider ID or display name.
24
25- `api key`: stores an API key in `$ZUT_HOME/auth.json` when the provider uses a normal key.
26- `subscription`: stores OAuth credentials for subscription-backed providers.
27
28Use `/logout` to remove stored credentials.
29
30Some providers need more than a single pasted key. For those providers,
31`/login` shows setup instructions instead of opening a localhost browser form.
32This avoids broken browser flows in SSH, containers, and `kubectl exec`
33sessions.
34
35### Command-backed API keys
36
37A provider can obtain its API key from a password manager or another local program. Configure this directly in `$ZUT_HOME/auth.json`:
38
39```json
40{
41  "openai": {
42    "api_key_command": {
43      "program": "op",
44      "args": ["read", "op://Work/OpenAI/credential"],
45      "timeout_ms": 120000
46    }
47  }
48}
49```
50
51Custom provider credentials use the same object under `additional_api_key_creds`:
52
53```json
54{
55  "additional_api_key_creds": {
56    "my-company": {
57      "api_key_command": {
58        "program": "secret-tool",
59        "args": ["read", "my-company-api-key"]
60      }
61    }
62  }
63}
64```
65
66The program is started directly rather than through a shell. The timeout defaults to 120 seconds. It must write one non-empty line to stdout, with no more than 64 KiB of output. Trailing CR/LF characters are removed. Successful results stay only in process memory and are reused until zut exits.
67
68Command-backed credentials count as logged in without being executed. zut materializes one only when selecting that provider; background model discovery skips it. `/login` with a normal key replaces the command, while `/logout` clears it. Because `auth.json` can cause program execution, keep it user-writable only and do not use files from untrusted sources.
69
70Setup-instruction providers:
71
72- Amazon Bedrock
73- Google Vertex AI
74- Cloudflare Workers AI
75- Cloudflare AI Gateway
76- Azure OpenAI Responses
77
78## Subscription providers
79
80These providers support subscription login:
81
82| Provider | Notes |
83| --- | --- |
84| Anthropic | Claude Pro/Max OAuth credentials. |
85| OpenAI Codex | ChatGPT Plus/Pro Codex subscription route. Separate from the OpenAI API-key provider. |
86| Kimi | Kimi subscription login. |
87| xAI | SuperGrok or X Premium device-code login. The browser URL is prefilled with the device code. |
88| GitHub Copilot | GitHub Copilot token flow. |
89
90OAuth tokens are stored in `$ZUT_HOME/auth.json` and refreshed when refresh is
91available.
92
93## API-key providers
94
95These providers can use environment variables. Simple API-key providers can
96also be configured through `/login`. Providers that require extra cloud setup
97show instructions and should be configured with environment variables.
98
99| Provider | Environment variable | Stored key |
100| --- | --- | --- |
101| Anthropic | `ANTHROPIC_API_KEY` | `anthropic` |
102| OpenAI | `OPENAI_API_KEY` | `openai` |
103| OpenAI Responses | `OPENAI_API_KEY` | `openai-responses` |
104| Kimi | `KIMI_API_KEY` or `MOONSHOT_API_KEY` | `kimi` |
105| Google Gemini | `GEMINI_API_KEY` or `GOOGLE_API_KEY` | `google` |
106| DeepSeek | `DEEPSEEK_API_KEY` | `deepseek` |
107| Ollama Cloud | `OLLAMA_API_KEY` | `ollama` |
108| Moonshot AI | `MOONSHOT_API_KEY` | `moonshotai` |
109| Moonshot AI China | `MOONSHOT_API_KEY` | `moonshotai-cn` |
110| Groq | `GROQ_API_KEY` | `groq` |
111| xAI | `XAI_API_KEY` | `xai` |
112| Cerebras | `CEREBRAS_API_KEY` | `cerebras` |
113| Together AI | `TOGETHER_API_KEY` | `together` |
114| Hugging Face | `HF_TOKEN` | `huggingface` |
115| OpenRouter | `OPENROUTER_API_KEY` | `openrouter` |
116| Mistral | `MISTRAL_API_KEY` | `mistral` |
117| ZAI | `ZAI_API_KEY` | `zai` |
118| Xiaomi MiMo | `XIAOMI_API_KEY` | `xiaomi` |
119| Xiaomi Token Plan Amsterdam | `XIAOMI_TOKEN_PLAN_AMS_API_KEY` | `xiaomi-token-plan-ams` |
120| Xiaomi Token Plan China | `XIAOMI_TOKEN_PLAN_CN_API_KEY` | `xiaomi-token-plan-cn` |
121| Xiaomi Token Plan Singapore | `XIAOMI_TOKEN_PLAN_SGP_API_KEY` | `xiaomi-token-plan-sgp` |
122| MiniMax | `MINIMAX_API_KEY` | `minimax` |
123| MiniMax China | `MINIMAX_CN_API_KEY` or `MINIMAX_API_KEY` | `minimax-cn` |
124| Fireworks | `FIREWORKS_API_KEY` | `fireworks` |
125| Vercel AI Gateway | `AI_GATEWAY_API_KEY` | `vercel-ai-gateway` |
126| OpenCode Zen | `OPENCODE_API_KEY` | `opencode` |
127| OpenCode Go | `OPENCODE_API_KEY` | `opencode-go` |
128| GitHub Copilot token | `COPILOT_GITHUB_TOKEN` or `GITHUB_COPILOT_TOKEN` | `github-copilot` |
129| Cloudflare Workers AI | `CLOUDFLARE_API_KEY` | `cloudflare-workers-ai` |
130| Cloudflare AI Gateway | `CLOUDFLARE_API_KEY` | `cloudflare-ai-gateway` |
131| Azure OpenAI Responses | `AZURE_OPENAI_API_KEY` | `azure-openai-responses` |
132
133Example:
134
135```bash
136export OPENROUTER_API_KEY=...
137zut --provider openrouter
138```
139
140### Ollama local vs Cloud
141
142The `ollama` provider serves both local and Cloud models over the OpenAI-compatible `/v1` API. Secrets are sent only as `Authorization: Bearer` headers, never in URLs or logs.
143
144- No key: `http://localhost:11434`, no auth (local models).
145- Real key (`--api-key`, `OLLAMA_API_KEY`, or `/login` → Ollama) with no explicit endpoint: `https://ollama.com/v1` (Cloud models).
146- `--base-url` always wins; a `baseUrl` in `models.json` (model- or provider-level) wins over the automatic default.
147
148### OpenCode Go model catalog
149
150OpenCode Go is synchronized instead of using a hand-maintained model list. When
151`OPENCODE_API_KEY` or a configured OpenCode Go credential is available, zut
152reads the model IDs published by
153[`https://opencode.ai/zen/go/v1/models`](https://opencode.ai/zen/go/v1/models)
154and enriches them with names, limits, prices, reasoning options, and adapter
155metadata from the [Models.dev JSON API](https://models.dev/api.json), which
156powers the [OpenCode Go provider page](https://models.dev/providers/opencode-go/).
157The published list is authoritative for the catalog and picker, but it is not
158an entitlement check: consent, regional eligibility, quota, and account policy
159can still reject an inference request.
160
161Models.dev adapter metadata selects the wire protocol for each model:
162`@ai-sdk/openai` uses OpenAI Responses, `@ai-sdk/openai-compatible` uses Chat
163Completions, and `@ai-sdk/anthropic` uses Anthropic Messages. If that metadata
164is unavailable, zut keeps the published model ID and uses only its narrow
165bootstrap fallback; protocol compatibility cannot be guaranteed until the
166metadata is available. An explicit unsupported adapter fails with an
167unsupported-protocol error instead of silently choosing Chat Completions.
168Explicit and discovered base URLs remain the credential-scoped gateway
169endpoint; public metadata does not redirect requests to another host.
170
171The merged catalog is cached in `$ZUT_HOME/models-cache.json` for 24 hours;
172OpenCode Go entries are scoped to both the credential fingerprint and endpoint.
173`--list-models` waits for a refresh when the cache is stale, while SDK runtimes
174load matching cached metadata before resolving. Synchronous refreshes resolve
175configured `api_key_command` credentials; unsolicited startup refreshes leave
176those commands untouched.
177
178## Fast mode
179
180Use `/fast` or `/settings` to enable **fast mode** for subsequent model calls.
181The setting is off by default and is stored as `fast_mode` in
182`$ZUT_HOME/config.json`.
183
184Fast mode currently requests OpenAI's Fast service tier for OpenAI Chat,
185OpenAI Responses, and OpenAI Codex requests. Enabling it while using another
186provider currently returns an error instead of silently changing that provider's
187request. Subagent children inherit the setting from their parent unless their
188profile sets `fastMode` explicitly.
189
190## GPT-6 Astra
191
192`gpt-6-astra` is available through `openai` (Responses API with an API key)
193and `openai-codex` (ChatGPT subscription). Both catalog entries deliberately
194use a **500,000-token working context budget**, rather than the larger advertised
195maximum, and retain a 128,000-token maximum output. Native reasoning efforts
196`low`, `medium`, `high`, `xhigh`, and `max` are supported. Selecting reasoning off
197omits the effort parameter; it does not disable Astra's reasoning.
198
199Standard API rates per million tokens are $10 input, $50 output, $1 cache reads,
200and $12.50 cache writes. Above **272,000 total input tokens**, including cache
201reads and writes, rates for the entire request become $20 input, $75 output,
202$2 cache reads, and $25 cache writes. The 500k working budget does not avoid
203this pricing tier. Codex catalog dollar estimates use these API-equivalent
204rates, not subscription billing or quota measurements.
205
206Sources: [Astra model specifications](https://developers.openai.com/api/docs/models/gpt-6-astra)
207and [prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching).
208
209## GPT-6.1 Sol
210
211`gpt-6.1-sol` is available through `openai` (Responses API) and `openai-codex`
212with the same 500,000-token working budget, 128,000-token maximum output, and
213native reasoning efforts `low` through `max` as Astra.
214
215Standard API rates per million tokens are $2 input, $10 output, $0.10 cache
216reads, and $2.50 cache writes. Above 272,000 total input tokens, the whole
217request is billed at $4 input, $15 output, $0.20 cache reads, and $5 cache
218writes.
219
220Source: [GPT-6.1 Sol model specifications](https://developers.openai.com/api/docs/models/gpt-6.1-sol).
221
222## Session identity and caching
223
224zut supplies every provider request with a root cache identity, a
225conversation-thread identity, and a per-prompt turn ID. They are opaque local
226correlations; adapters translate them only when their exact provider, model,
227and endpoint declaration supports it. Resident children share the root cache
228identity but retain distinct thread identities. The OpenAI/Codex Responses
229route uses the root identity as `prompt_cache_key`; declared Codex routes also
230receive their documented session and thread routing headers. OpenCode Go
231requests identify zut with `x-opencode-client` and `User-Agent`, use the stable
232thread (or root cache) ID as `x-opencode-session`, and use the turn ID as
233`x-opencode-request`. A generic OpenAI-compatible endpoint never receives
234these extensions merely because it accepts an OpenAI-shaped request. JSON
235event mode also reports sanitized cache diagnostics (`eligible`, `mode`,
236`transport`, and `continuation`). `eligible`
237is a conservative local prompt-length estimate; OpenAI performs the authoritative
238tokenization and cache decision. These records never include prompts, durable
239IDs, request bodies, or credentials.
240
241When a provider reports cache detail, zut displays cache-read hit rate as
242`cache reads / cache-reporting prompt input`. A shown `0%` is a confirmed
243cache miss; no percentage means the provider or historical session did not
244report cache detail. Cache writes remain separate from that hit rate. Provider
245retention and eviction, request routing, and OpenAI's 1,024-token eligibility
246minimum can still produce a miss even when the stable prefix is unchanged.
247
248On the official public OpenAI Responses endpoint, GPT-5.6 and GPT-6 Astra
249requests include an explicit cache breakpoint after the stable instruction/tool
250prefix while retaining implicit caching for conversation history. The Codex
251subscription route and custom Responses endpoints do not receive these explicit
252cache controls. Changing request-level reasoning effort can still disrupt prefix
253reuse: zut does not yet send cache-preserving `configuration_update` items.
254
255### OpenAI Responses WebSocket mode
256
257For the exact public endpoint `https://api.openai.com/v1/responses`, zut uses
258the Responses API WebSocket mode for the OpenAI Responses route. It keeps one
259connection per conversation thread, sends one `response.create` at a time,
260and continues the most recent completed response with `previous_response_id`
261only when the complete response-context configuration and prior input prefix
262match exactly, including the completed assistant output that precedes a new
263suffix. Compatible continuations send only new tool or user input.
264The request uses `store:false`; its only provider-specific handshake header is
265`Authorization: Bearer ...`, never the
266ChatGPT/Codex account, routing, or session headers.
267
268The capability is deliberately not inherited by `--base-url`, compatible
269providers, Azure, or the ChatGPT Codex subscription endpoint. Those clients
270continue to use their declared HTTP/SSE contracts. A request without a logical
271session ID, a failed WebSocket setup,
272an injected nonstandard RoundTripper, or a configured proxy falls back to
273HTTP/SSE with the same logical session/cache identity. Cancellation closes
274only the active session socket; a normally completed turn keeps it warm for
275the next continuation.
276
277### Responses explicit continuation
278
279When a Responses completion carries an explicit `end_turn:false`, zut treats the finished inference as a continuation (`turn_end` with `stop:"continue"`) and issues another model call in the same turn, reusing its request context, guards, usage accounting, and queued-input boundary. Tool calls still take precedence over the signal, and errors or cancellation never become continuations. A missing, null, or true `end_turn` keeps the existing behavior and ends the turn.
280
281### Responses reasoning replay
282
283You can switch models or providers within the same session. Assistant text and tool exchanges remain available; encrypted reasoning and thought signatures from a different provider or model are omitted from outgoing requests without editing session history. Switching back allows the matching model to replay its own metadata.
284
285Responses requests run with `store:false`, so only reasoning blocks with an encrypted payload and a valid Responses item ID (when present) are replayed. Session loading recovers message provenance from the provider/model metadata recorded alongside turns. Older compacted messages without provenance retain best-effort replay; malformed reasoning IDs are omitted.
286
287## Local llama.cpp router
288
289The `llama.cpp` provider connects to a multi-model router and is separate from the `ollama` provider. Ollama normally uses `http://localhost:11434`; entering that URL for llama.cpp management produces a 404 because Ollama does not implement the router endpoints.
290
291Install a current llama.cpp build. On macOS:
292
293```bash
294brew install llama.cpp
295# use `brew upgrade llama.cpp` for an existing installation
296```
297
298Start `llama-server` without `--model`, `-m`, or `-hf`. Those flags select one model and disable the router behavior zut expects.
299
300```bash
301mkdir -p ~/llama-models
302
303llama-server \
304  --models-dir ~/llama-models \
305  --no-models-autoload \
306  --jinja \
307  --host 127.0.0.1 \
308  --port 8080 \
309  -ngl 999 \
310  -c 32768
311```
312
313This configuration discovers GGUF files below `~/llama-models`, leaves model loading under explicit user control, enables chat templates, requests GPU offload, and sets a 32K context. Tune the GPU layers and context size for the available memory.
314
315Check the management API before opening zut:
316
317```bash
318curl http://127.0.0.1:8080/health
319curl http://127.0.0.1:8080/models
320```
321
322`/models` must return JSON with a `data` array. A 404 indicates the wrong URL, an older server build, or single-model mode.
323
324To save the connection, run `/login`, choose `api key`, and select `llama.cpp`. Enter `http://127.0.0.1:8080`, without `/v1`, followed by an optional bearer key. zut validates the URL and stores both values in `$ZUT_HOME/auth.json`. Environment variables override the saved connection:
325
326```bash
327export LLAMA_BASE_URL=http://127.0.0.1:8080
328export LLAMA_API_KEY=optional-secret
329```
330
331If a key is configured, start the router with the same `--api-key` value. A server bound only to `127.0.0.1` normally does not need authentication.
332
333After login, `/llama` becomes visible. It shows model state, searches Hugging Face for GGUF repositories, offers available quantizations, reports download and load progress, and explicitly loads or unloads models. Press `d` on a router-downloaded cache model to ask the router to remove it after confirmation. Models discovered through `--models-dir` or a preset are not removable through the router API and must be deleted from their configured source. Some llama.cpp versions retain shared Hugging Face repository artifacts such as `mmproj` files after removing the selected GGUF; remove that repository's cache directory manually if those artifacts are no longer needed. `HF_TOKEN` raises Hugging Face API limits and may be required for repository metadata. For gated files, approve access first and provide the authorized token to the `llama-server` process because the router performs the download.
334
335Opening `/model` refreshes the router and displays its loaded models under provider `llama.cpp`. Unloaded models cannot handle inference and therefore remain exclusive to `/llama` until loaded. Inference is sent to the router's derived `/v1` endpoint.
336
337Ollama downloads are stored in Ollama's internal layout and do not automatically become llama.cpp models. Obtain a GGUF copy through `/llama` or place GGUF files under `~/llama-models`, then restart the router after adding files manually.
338
339## Cloud providers
340
341### Amazon Bedrock
342
343Bedrock is configured with AWS credentials, not a generic zut API-key entry.
344Use one of these credential sources:
345
346```bash
347# AWS profile
348export AWS_PROFILE=your-profile
349
350# IAM access keys
351export AWS_ACCESS_KEY_ID=AKIA...
352export AWS_SECRET_ACCESS_KEY=...
353export AWS_SESSION_TOKEN=... # only for temporary credentials
354
355# Bedrock API key bearer token
356export AWS_BEARER_TOKEN_BEDROCK=bedrock-api-key-...
357
358# Region
359export AWS_REGION=us-east-1
360```
361
362ECS task roles, IRSA, and other AWS SDK credential-chain sources are also
363supported.
364
365Example:
366
367```bash
368AWS_BEARER_TOKEN_BEDROCK=bedrock-api-key-... AWS_REGION=us-east-1 \
369  zut --provider amazon-bedrock --model anthropic.claude-sonnet-4-5-20250929-v1:0
370```
371
372Some Bedrock models require regional inference-profile IDs for on-demand
373throughput, such as `us.` or `eu.` prefixed model IDs. zut rewrites known
374families automatically where possible. Claude Opus 5 uses
375`global.anthropic.claude-opus-5`; zut maps the bare foundation-model ID to that
376global profile and supports its adaptive reasoning levels. Explicit profile IDs
377and ARNs are left unchanged.
378
379### Google Vertex AI
380
381Vertex can use a Google API key when available:
382
383```bash
384export GOOGLE_CLOUD_API_KEY=...
385zut --provider google-vertex
386```
387
388For service-account or application-default credentials, set the standard
389Google environment variables used by your deployment.
390
391### Cloudflare AI Gateway
392
393Cloudflare AI Gateway needs a Cloudflare token plus account and gateway IDs:
394
395```bash
396export CLOUDFLARE_API_KEY=...
397export CLOUDFLARE_ACCOUNT_ID=...
398export CLOUDFLARE_GATEWAY_ID=...
399zut --provider cloudflare-ai-gateway
400```
401
402### Cloudflare Workers AI
403
404Workers AI needs a Cloudflare token and account ID:
405
406```bash
407export CLOUDFLARE_API_KEY=...
408export CLOUDFLARE_ACCOUNT_ID=...
409zut --provider cloudflare-workers-ai
410```
411
412### Azure OpenAI Responses
413
414```bash
415export AZURE_OPENAI_API_KEY=...
416export AZURE_OPENAI_BASE_URL=https://your-resource.openai.azure.com
417export AZURE_OPENAI_API_VERSION=v1 # optional, v1 is the default
418zut --provider azure-openai-responses
419```
420
421The provider uses Azure's Responses API. If deployment names differ from zut
422model IDs, map them without changing the catalog:
423
424```bash
425export AZURE_OPENAI_DEPLOYMENT_NAME_MAP='gpt-5.6-luna=luna-preview,gpt-5.6-sol=sol-preview'
426```
427
428## Auth file
429
430Credentials are stored in `$ZUT_HOME/auth.json` with user-only permissions
431when zut creates the file.
432
433Example:
434
435```json
436{
437  "anthropic": { "api_key": "sk-ant-..." },
438  "openai": { "api_key": "sk-..." },
439  "google": { "api_key": "..." },
440  "additional_api_key_creds": {
441    "openrouter": { "api_key": "..." },
442    "mistral": { "api_key": "..." }
443  }
444}
445```
446
447The top-level keys are used for providers with dedicated credential fields.
448Other API-key providers are stored under `additional_api_key_creds`. Prefer
449`/login` so zut writes the correct schema.
450
451## Custom providers and models
452
453Use `$ZUT_HOME/models.json` for private models, deployment aliases, local
454servers, or OpenAI-compatible gateways that are not in the built-in catalog.
455User entries override built-in entries with the same provider and model ID, and
456adding a `models.json` no longer hides the built-in catalog: your entries are
457merged on top of the baked-in and live-discovered models.
458
459A top-level provider key that is not a built-in id defines a custom provider.
460Give it a provider-level `baseUrl` and an `api` wire format (`openai` for
461OpenAI-compatible Chat Completions, the default, `openai-responses` for the
462OpenAI Responses API, or `anthropic` for the Anthropic Messages API). A
463model-level `baseUrl` overrides the provider-level one for that model, and an
464unknown `api` value falls back to `openai` with a warning.
465
466```json
467{
468  "providers": {
469    "my-company": {
470      "baseUrl": "https://llm.mycompany.com/v1",
471      "api": "openai",
472      "models": [
473        { "id": "company-llm-v2", "name": "Company LLM v2" }
474      ]
475    }
476  }
477}
478```
479
480Custom providers are first-class: they appear in `--list-models`, `/model`, and
481`/login`. `models.json` never stores secrets. Supply the key through `/login`,
482`--api-key`, or a derived environment variable in upper snake case, so
483`my-company` reads `MY_COMPANY_API_KEY`. Because many self-hosted gateways do
484not expose a model-list endpoint, custom provider keys are accepted and stored
485without a verification probe; an invalid key surfaces on the first model call.
486
487To retrieve this custom provider's key from a password manager, add a matching
488entry to `$ZUT_HOME/auth.json`:
489
490```json
491{
492  "additional_api_key_creds": {
493    "my-company": {
494      "api_key_command": {
495        "program": "op",
496        "args": ["read", "op://Work/OpenAI/credential"],
497        "timeout_ms": 120000
498      }
499    }
500  }
501}
502```
503
504The provider IDs in `models.json` and `auth.json` must match. Then select the
505custom model directly:
506
507```sh
508zut --provider my-company --model company-llm-v2
509```
510
511## Credential resolution
512
513For each request, zut checks credentials in this order:
514
5151. Explicit CLI key, such as `--api-key`.
5162. Provider-specific environment variables (including derived custom-provider
517   variables such as `MY_COMPANY_API_KEY`).
5183. `$ZUT_HOME/auth.json`, including custom provider keys saved by `/login`.
519
520`models.json` itself never stores credentials; it only describes models and
521their endpoints.
522
523Bedrock then uses the AWS SDK credential chain for the actual request.