providers.md Raw History Folder

zut providers

zut ships with built-in providers and a model catalog. You can select models with /model, list them with zut --list-models, and add private models in $ZUT_HOME/models.json.

HTTP proxy

Set one global proxy for zut-managed HTTP and HTTPS traffic with the http_proxy key in $ZUT_HOME/config.json:

{
  "http_proxy": "http://127.0.0.1:7890"
}

zut applies this setting at startup. Existing HTTP_PROXY, HTTPS_PROXY, http_proxy, and https_proxy environment variables take precedence for their corresponding protocol. Standard NO_PROXY and no_proxy bypass lists remain effective. Restart zut after changing the setting.

config.json is not a credential store. If the proxy URL contains a username or password, prefer setting the proxy environment variables in a protected shell or service configuration rather than saving those credentials in config.json.

Use /login in interactive mode. Type in either provider picker to filter the list by provider ID or display name.

api key: stores an API key in $ZUT_HOME/auth.json when the provider uses a normal key.
subscription: stores OAuth credentials for subscription-backed providers.

Use /logout to remove stored credentials.

Some providers need more than a single pasted key. For those providers, /login shows setup instructions instead of opening a localhost browser form. This avoids broken browser flows in SSH, containers, and kubectl exec sessions.

Command-backed API keys

A provider can obtain its API key from a password manager or another local program. Configure this directly in $ZUT_HOME/auth.json:

{
  "openai": {
    "api_key_command": {
      "program": "op",
      "args": ["read", "op://Work/OpenAI/credential"],
      "timeout_ms": 120000
    }
  }
}

Custom provider credentials use the same object under additional_api_key_creds:

{
  "additional_api_key_creds": {
    "my-company": {
      "api_key_command": {
        "program": "secret-tool",
        "args": ["read", "my-company-api-key"]
      }
    }
  }
}

The program is started directly rather than through a shell. The timeout defaults to 120 seconds. It must write one non-empty line to stdout, with no more than 64 KiB of output. Trailing CR/LF characters are removed. Successful results stay only in process memory and are reused until zut exits.

Command-backed credentials count as logged in without being executed. zut materializes one only when selecting that provider; background model discovery skips it. /login with a normal key replaces the command, while /logout clears it. Because auth.json can cause program execution, keep it user-writable only and do not use files from untrusted sources.

Setup-instruction providers:

Amazon Bedrock
Google Vertex AI
Cloudflare Workers AI
Cloudflare AI Gateway
Azure OpenAI Responses

Subscription providers

These providers support subscription login:

Provider	Notes
Anthropic	Claude Pro/Max OAuth credentials.
OpenAI Codex	ChatGPT Plus/Pro Codex subscription route. Separate from the OpenAI API-key provider.
Kimi	Kimi subscription login.
xAI	SuperGrok or X Premium device-code login. The browser URL is prefilled with the device code.
GitHub Copilot	GitHub Copilot token flow.

OAuth tokens are stored in $ZUT_HOME/auth.json and refreshed when refresh is available.

API-key providers

These providers can use environment variables. Simple API-key providers can also be configured through /login. Providers that require extra cloud setup show instructions and should be configured with environment variables.

Provider	Environment variable	Stored key
Anthropic	`ANTHROPIC_API_KEY`	`anthropic`
OpenAI	`OPENAI_API_KEY`	`openai`
OpenAI Responses	`OPENAI_API_KEY`	`openai-responses`
Kimi	`KIMI_API_KEY` or `MOONSHOT_API_KEY`	`kimi`
Google Gemini	`GEMINI_API_KEY` or `GOOGLE_API_KEY`	`google`
DeepSeek	`DEEPSEEK_API_KEY`	`deepseek`
Moonshot AI	`MOONSHOT_API_KEY`	`moonshotai`
Moonshot AI China	`MOONSHOT_API_KEY`	`moonshotai-cn`
Groq	`GROQ_API_KEY`	`groq`
xAI	`XAI_API_KEY`	`xai`
Cerebras	`CEREBRAS_API_KEY`	`cerebras`
Together AI	`TOGETHER_API_KEY`	`together`
Hugging Face	`HF_TOKEN`	`huggingface`
OpenRouter	`OPENROUTER_API_KEY`	`openrouter`
Mistral	`MISTRAL_API_KEY`	`mistral`
ZAI	`ZAI_API_KEY`	`zai`
Xiaomi MiMo	`XIAOMI_API_KEY`	`xiaomi`
Xiaomi Token Plan Amsterdam	`XIAOMI_TOKEN_PLAN_AMS_API_KEY`	`xiaomi-token-plan-ams`
Xiaomi Token Plan China	`XIAOMI_TOKEN_PLAN_CN_API_KEY`	`xiaomi-token-plan-cn`
Xiaomi Token Plan Singapore	`XIAOMI_TOKEN_PLAN_SGP_API_KEY`	`xiaomi-token-plan-sgp`
MiniMax	`MINIMAX_API_KEY`	`minimax`
MiniMax China	`MINIMAX_CN_API_KEY` or `MINIMAX_API_KEY`	`minimax-cn`
Fireworks	`FIREWORKS_API_KEY`	`fireworks`
Vercel AI Gateway	`AI_GATEWAY_API_KEY`	`vercel-ai-gateway`
OpenCode Zen	`OPENCODE_API_KEY`	`opencode`
OpenCode Go	`OPENCODE_API_KEY`	`opencode-go`
GitHub Copilot token	`COPILOT_GITHUB_TOKEN` or `GITHUB_COPILOT_TOKEN`	`github-copilot`
Cloudflare Workers AI	`CLOUDFLARE_API_KEY`	`cloudflare-workers-ai`
Cloudflare AI Gateway	`CLOUDFLARE_API_KEY`	`cloudflare-ai-gateway`
Azure OpenAI Responses	`AZURE_OPENAI_API_KEY`	`azure-openai-responses`

Example:

export OPENROUTER_API_KEY=...
zut --provider openrouter

Fast mode

Use /fast or /settings to enable fast mode for subsequent model calls. The setting is off by default and is stored as fast_mode in $ZUT_HOME/config.json.

Fast mode currently requests OpenAI's Fast service tier for OpenAI Chat, OpenAI Responses, and OpenAI Codex requests. Enabling it while using another provider currently returns an error instead of silently changing that provider's request. Subagent children inherit the setting from their parent unless their profile sets fastMode explicitly.

GPT-6 Astra

gpt-6-astra is available through openai (Responses API with an API key) and openai-codex (ChatGPT subscription). Both catalog entries deliberately use a 500,000-token working context budget, rather than the larger advertised maximum, and retain a 128,000-token maximum output. Native reasoning efforts low, medium, high, xhigh, and max are supported. Selecting reasoning off omits the effort parameter; it does not disable Astra's reasoning.

Standard API rates per million tokens are $10 input, $50 output, $1 cache reads, and $12.50 cache writes. Above 272,000 total input tokens, including cache reads and writes, rates for the entire request become $20 input, $75 output, $2 cache reads, and $25 cache writes. The 500k working budget does not avoid this pricing tier. Codex catalog dollar estimates use these API-equivalent rates, not subscription billing or quota measurements.

Sources: Astra model specifications and prompt caching.

Session identity and caching

zut supplies every provider request with a root cache identity, a conversation-thread identity, and a per-prompt turn ID. They are opaque local correlations; adapters translate them only when their exact provider, model, and endpoint declaration supports it. Resident children share the root cache identity but retain distinct thread identities. The OpenAI/Codex Responses route uses the root identity as prompt_cache_key; declared Codex routes also receive their documented session and thread routing headers. A generic OpenAI-compatible endpoint never receives these extensions merely because it accepts an OpenAI-shaped request. JSON event mode also reports sanitized cache diagnostics (eligible, mode, transport, and continuation). eligible is a conservative local prompt-length estimate; OpenAI performs the authoritative tokenization and cache decision. These records never include prompts, durable IDs, request bodies, or credentials.

When a provider reports cache detail, zut displays cache-read hit rate as cache reads / cache-reporting prompt input. A shown 0% is a confirmed cache miss; no percentage means the provider or historical session did not report cache detail. Cache writes remain separate from that hit rate. Provider retention and eviction, request routing, and OpenAI's 1,024-token eligibility minimum can still produce a miss even when the stable prefix is unchanged.

On the official public OpenAI Responses endpoint, GPT-5.6 and GPT-6 Astra requests include an explicit cache breakpoint after the stable instruction/tool prefix while retaining implicit caching for conversation history. The Codex subscription route and custom Responses endpoints do not receive these explicit cache controls. Changing request-level reasoning effort can still disrupt prefix reuse: zut does not yet send cache-preserving configuration_update items.

OpenAI Responses WebSocket mode

For the exact public endpoint https://api.openai.com/v1/responses, zut uses the Responses API WebSocket mode for the OpenAI Responses route. It keeps one connection per conversation thread, sends one response.create at a time, and continues the most recent completed response with previous_response_id only when the complete response-context configuration and prior input prefix match exactly, including the completed assistant output that precedes a new suffix. Compatible continuations send only new tool or user input. The request uses store:false; its only provider-specific handshake header is Authorization: Bearer ..., never the ChatGPT/Codex account, routing, or session headers.

The capability is deliberately not inherited by --base-url, compatible providers, Azure, or the ChatGPT Codex subscription endpoint. Those clients continue to use their declared HTTP/SSE contracts. A request without a logical session ID, a failed WebSocket setup, an injected nonstandard RoundTripper, or a configured proxy falls back to HTTP/SSE with the same logical session/cache identity. Cancellation closes only the active session socket; a normally completed turn keeps it warm for the next continuation.

Local llama.cpp router

The llama.cpp provider connects to a multi-model router and is separate from the ollama provider. Ollama normally uses http://localhost:11434; entering that URL for llama.cpp management produces a 404 because Ollama does not implement the router endpoints.

Install a current llama.cpp build. On macOS:

brew install llama.cpp
# use `brew upgrade llama.cpp` for an existing installation

Start llama-server without --model, -m, or -hf. Those flags select one model and disable the router behavior zut expects.

mkdir -p ~/llama-models

llama-server \
  --models-dir ~/llama-models \
  --no-models-autoload \
  --jinja \
  --host 127.0.0.1 \
  --port 8080 \
  -ngl 999 \
  -c 32768

This configuration discovers GGUF files below ~/llama-models, leaves model loading under explicit user control, enables chat templates, requests GPU offload, and sets a 32K context. Tune the GPU layers and context size for the available memory.

Check the management API before opening zut:

curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/models

/models must return JSON with a data array. A 404 indicates the wrong URL, an older server build, or single-model mode.

To save the connection, run /login, choose api key, and select llama.cpp. Enter http://127.0.0.1:8080, without /v1, followed by an optional bearer key. zut validates the URL and stores both values in $ZUT_HOME/auth.json. Environment variables override the saved connection:

export LLAMA_BASE_URL=http://127.0.0.1:8080
export LLAMA_API_KEY=optional-secret

If a key is configured, start the router with the same --api-key value. A server bound only to 127.0.0.1 normally does not need authentication.

After login, /llama becomes visible. It shows model state, searches Hugging Face for GGUF repositories, offers available quantizations, reports download and load progress, and explicitly loads or unloads models. Press d on a router-downloaded cache model to ask the router to remove it after confirmation. Models discovered through --models-dir or a preset are not removable through the router API and must be deleted from their configured source. Some llama.cpp versions retain shared Hugging Face repository artifacts such as mmproj files after removing the selected GGUF; remove that repository's cache directory manually if those artifacts are no longer needed. HF_TOKEN raises Hugging Face API limits and may be required for repository metadata. For gated files, approve access first and provide the authorized token to the llama-server process because the router performs the download.

Opening /model refreshes the router and displays its loaded models under provider llama.cpp. Unloaded models cannot handle inference and therefore remain exclusive to /llama until loaded. Inference is sent to the router's derived /v1 endpoint.

Ollama downloads are stored in Ollama's internal layout and do not automatically become llama.cpp models. Obtain a GGUF copy through /llama or place GGUF files under ~/llama-models, then restart the router after adding files manually.

Cloud providers

Amazon Bedrock

Bedrock is configured with AWS credentials, not a generic zut API-key entry. Use one of these credential sources:

# AWS profile
export AWS_PROFILE=your-profile

# IAM access keys
export AWS_ACCESS_KEY_ID=AKIA...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SESSION_TOKEN=... # only for temporary credentials

# Bedrock API key bearer token
export AWS_BEARER_TOKEN_BEDROCK=bedrock-api-key-...

# Region
export AWS_REGION=us-east-1

ECS task roles, IRSA, and other AWS SDK credential-chain sources are also supported.

Example:

AWS_BEARER_TOKEN_BEDROCK=bedrock-api-key-... AWS_REGION=us-east-1 \
  zut --provider amazon-bedrock --model anthropic.claude-sonnet-4-5-20250929-v1:0

Some Bedrock models require regional inference-profile IDs for on-demand throughput, such as us. or eu. prefixed model IDs. zut rewrites known families automatically where possible. Claude Opus 5 uses global.anthropic.claude-opus-5; zut maps the bare foundation-model ID to that global profile and supports its adaptive reasoning levels. Explicit profile IDs and ARNs are left unchanged.

Google Vertex AI

Vertex can use a Google API key when available:

export GOOGLE_CLOUD_API_KEY=...
zut --provider google-vertex

For service-account or application-default credentials, set the standard Google environment variables used by your deployment.

Cloudflare AI Gateway

Cloudflare AI Gateway needs a Cloudflare token plus account and gateway IDs:

export CLOUDFLARE_API_KEY=...
export CLOUDFLARE_ACCOUNT_ID=...
export CLOUDFLARE_GATEWAY_ID=...
zut --provider cloudflare-ai-gateway

Cloudflare Workers AI

Workers AI needs a Cloudflare token and account ID:

export CLOUDFLARE_API_KEY=...
export CLOUDFLARE_ACCOUNT_ID=...
zut --provider cloudflare-workers-ai

Azure OpenAI Responses

export AZURE_OPENAI_API_KEY=...
export AZURE_OPENAI_BASE_URL=https://your-resource.openai.azure.com
export AZURE_OPENAI_API_VERSION=v1 # optional, v1 is the default
zut --provider azure-openai-responses

The provider uses Azure's Responses API. If deployment names differ from zut model IDs, map them without changing the catalog:

export AZURE_OPENAI_DEPLOYMENT_NAME_MAP='gpt-5.6-luna=luna-preview,gpt-5.6-sol=sol-preview'

Auth file

Credentials are stored in $ZUT_HOME/auth.json with user-only permissions when zut creates the file.

Example:

{
  "anthropic": { "api_key": "sk-ant-..." },
  "openai": { "api_key": "sk-..." },
  "google": { "api_key": "..." },
  "additional_api_key_creds": {
    "openrouter": { "api_key": "..." },
    "mistral": { "api_key": "..." }
  }
}

The top-level keys are used for providers with dedicated credential fields. Other API-key providers are stored under additional_api_key_creds. Prefer /login so zut writes the correct schema.

Custom providers and models

Use $ZUT_HOME/models.json for private models, deployment aliases, local servers, or OpenAI-compatible gateways that are not in the built-in catalog. User entries override built-in entries with the same provider and model ID, and adding a models.json no longer hides the built-in catalog: your entries are merged on top of the baked-in and live-discovered models.

A top-level provider key that is not a built-in id defines a custom provider. Give it a provider-level baseUrl and an api wire format (openai for OpenAI-compatible Chat Completions, the default, openai-responses for the OpenAI Responses API, or anthropic for the Anthropic Messages API). A model-level baseUrl overrides the provider-level one for that model, and an unknown api value falls back to openai with a warning.

{
  "providers": {
    "my-company": {
      "baseUrl": "https://llm.mycompany.com/v1",
      "api": "openai",
      "models": [
        { "id": "company-llm-v2", "name": "Company LLM v2" }
      ]
    }
  }
}

Custom providers are first-class: they appear in --list-models, /model, and /login. models.json never stores secrets. Supply the key through /login, --api-key, or a derived environment variable in upper snake case, so my-company reads MY_COMPANY_API_KEY. Because many self-hosted gateways do not expose a model-list endpoint, custom provider keys are accepted and stored without a verification probe; an invalid key surfaces on the first model call.

To retrieve this custom provider's key from a password manager, add a matching entry to $ZUT_HOME/auth.json:

{
  "additional_api_key_creds": {
    "my-company": {
      "api_key_command": {
        "program": "op",
        "args": ["read", "op://Work/OpenAI/credential"],
        "timeout_ms": 120000
      }
    }
  }
}

The provider IDs in models.json and auth.json must match. Then select the custom model directly:

zut --provider my-company --model company-llm-v2

Credential resolution

For each request, zut checks credentials in this order:

Explicit CLI key, such as --api-key.
Provider-specific environment variables (including derived custom-provider variables such as MY_COMPANY_API_KEY).
$ZUT_HOME/auth.json, including custom provider keys saved by /login.

models.json itself never stores credentials; it only describes models and their endpoints.

Bedrock then uses the AWS SDK credential chain for the actual request.

source

1# zut providers
2
3zut ships with built-in providers and a model catalog. You can select models
4with `/model`, list them with `zut --list-models`, and add private models in
5`$ZUT_HOME/models.json`.
6
7## HTTP proxy
8
9Set one global proxy for zut-managed HTTP and HTTPS traffic with the `http_proxy` key in `$ZUT_HOME/config.json`:
10
11```json
12{
13  "http_proxy": "http://127.0.0.1:7890"
14}
15```
16
17zut applies this setting at startup. Existing `HTTP_PROXY`, `HTTPS_PROXY`, `http_proxy`, and `https_proxy` environment variables take precedence for their corresponding protocol. Standard `NO_PROXY` and `no_proxy` bypass lists remain effective. Restart zut after changing the setting.
18
19`config.json` is not a credential store. If the proxy URL contains a username or password, prefer setting the proxy environment variables in a protected shell or service configuration rather than saving those credentials in `config.json`.
20
21## Login methods
22
23Use `/login` in interactive mode. Type in either provider picker to filter the list by provider ID or display name.
24
25- `api key`: stores an API key in `$ZUT_HOME/auth.json` when the provider uses a normal key.
26- `subscription`: stores OAuth credentials for subscription-backed providers.
27
28Use `/logout` to remove stored credentials.
29
30Some providers need more than a single pasted key. For those providers,
31`/login` shows setup instructions instead of opening a localhost browser form.
32This avoids broken browser flows in SSH, containers, and `kubectl exec`
33sessions.
34
35### Command-backed API keys
36
37A provider can obtain its API key from a password manager or another local program. Configure this directly in `$ZUT_HOME/auth.json`:
38
39```json
40{
41  "openai": {
42    "api_key_command": {
43      "program": "op",
44      "args": ["read", "op://Work/OpenAI/credential"],
45      "timeout_ms": 120000
46    }
47  }
48}
49```
50
51Custom provider credentials use the same object under `additional_api_key_creds`:
52
53```json
54{
55  "additional_api_key_creds": {
56    "my-company": {
57      "api_key_command": {
58        "program": "secret-tool",
59        "args": ["read", "my-company-api-key"]
60      }
61    }
62  }
63}
64```
65
66The program is started directly rather than through a shell. The timeout defaults to 120 seconds. It must write one non-empty line to stdout, with no more than 64 KiB of output. Trailing CR/LF characters are removed. Successful results stay only in process memory and are reused until zut exits.
67
68Command-backed credentials count as logged in without being executed. zut materializes one only when selecting that provider; background model discovery skips it. `/login` with a normal key replaces the command, while `/logout` clears it. Because `auth.json` can cause program execution, keep it user-writable only and do not use files from untrusted sources.
69
70Setup-instruction providers:
71
72- Amazon Bedrock
73- Google Vertex AI
74- Cloudflare Workers AI
75- Cloudflare AI Gateway
76- Azure OpenAI Responses
77
78## Subscription providers
79
80These providers support subscription login:
81
82| Provider | Notes |
83| --- | --- |
84| Anthropic | Claude Pro/Max OAuth credentials. |
85| OpenAI Codex | ChatGPT Plus/Pro Codex subscription route. Separate from the OpenAI API-key provider. |
86| Kimi | Kimi subscription login. |
87| xAI | SuperGrok or X Premium device-code login. The browser URL is prefilled with the device code. |
88| GitHub Copilot | GitHub Copilot token flow. |
89
90OAuth tokens are stored in `$ZUT_HOME/auth.json` and refreshed when refresh is
91available.
92
93## API-key providers
94
95These providers can use environment variables. Simple API-key providers can
96also be configured through `/login`. Providers that require extra cloud setup
97show instructions and should be configured with environment variables.
98
99| Provider | Environment variable | Stored key |
100| --- | --- | --- |
101| Anthropic | `ANTHROPIC_API_KEY` | `anthropic` |
102| OpenAI | `OPENAI_API_KEY` | `openai` |
103| OpenAI Responses | `OPENAI_API_KEY` | `openai-responses` |
104| Kimi | `KIMI_API_KEY` or `MOONSHOT_API_KEY` | `kimi` |
105| Google Gemini | `GEMINI_API_KEY` or `GOOGLE_API_KEY` | `google` |
106| DeepSeek | `DEEPSEEK_API_KEY` | `deepseek` |
107| Moonshot AI | `MOONSHOT_API_KEY` | `moonshotai` |
108| Moonshot AI China | `MOONSHOT_API_KEY` | `moonshotai-cn` |
109| Groq | `GROQ_API_KEY` | `groq` |
110| xAI | `XAI_API_KEY` | `xai` |
111| Cerebras | `CEREBRAS_API_KEY` | `cerebras` |
112| Together AI | `TOGETHER_API_KEY` | `together` |
113| Hugging Face | `HF_TOKEN` | `huggingface` |
114| OpenRouter | `OPENROUTER_API_KEY` | `openrouter` |
115| Mistral | `MISTRAL_API_KEY` | `mistral` |
116| ZAI | `ZAI_API_KEY` | `zai` |
117| Xiaomi MiMo | `XIAOMI_API_KEY` | `xiaomi` |
118| Xiaomi Token Plan Amsterdam | `XIAOMI_TOKEN_PLAN_AMS_API_KEY` | `xiaomi-token-plan-ams` |
119| Xiaomi Token Plan China | `XIAOMI_TOKEN_PLAN_CN_API_KEY` | `xiaomi-token-plan-cn` |
120| Xiaomi Token Plan Singapore | `XIAOMI_TOKEN_PLAN_SGP_API_KEY` | `xiaomi-token-plan-sgp` |
121| MiniMax | `MINIMAX_API_KEY` | `minimax` |
122| MiniMax China | `MINIMAX_CN_API_KEY` or `MINIMAX_API_KEY` | `minimax-cn` |
123| Fireworks | `FIREWORKS_API_KEY` | `fireworks` |
124| Vercel AI Gateway | `AI_GATEWAY_API_KEY` | `vercel-ai-gateway` |
125| OpenCode Zen | `OPENCODE_API_KEY` | `opencode` |
126| OpenCode Go | `OPENCODE_API_KEY` | `opencode-go` |
127| GitHub Copilot token | `COPILOT_GITHUB_TOKEN` or `GITHUB_COPILOT_TOKEN` | `github-copilot` |
128| Cloudflare Workers AI | `CLOUDFLARE_API_KEY` | `cloudflare-workers-ai` |
129| Cloudflare AI Gateway | `CLOUDFLARE_API_KEY` | `cloudflare-ai-gateway` |
130| Azure OpenAI Responses | `AZURE_OPENAI_API_KEY` | `azure-openai-responses` |
131
132Example:
133
134```bash
135export OPENROUTER_API_KEY=...
136zut --provider openrouter
137```
138
139## Fast mode
140
141Use `/fast` or `/settings` to enable **fast mode** for subsequent model calls.
142The setting is off by default and is stored as `fast_mode` in
143`$ZUT_HOME/config.json`.
144
145Fast mode currently requests OpenAI's Fast service tier for OpenAI Chat,
146OpenAI Responses, and OpenAI Codex requests. Enabling it while using another
147provider currently returns an error instead of silently changing that provider's
148request. Subagent children inherit the setting from their parent unless their
149profile sets `fastMode` explicitly.
150
151## GPT-6 Astra
152
153`gpt-6-astra` is available through `openai` (Responses API with an API key)
154and `openai-codex` (ChatGPT subscription). Both catalog entries deliberately
155use a **500,000-token working context budget**, rather than the larger advertised
156maximum, and retain a 128,000-token maximum output. Native reasoning efforts
157`low`, `medium`, `high`, `xhigh`, and `max` are supported. Selecting reasoning off
158omits the effort parameter; it does not disable Astra's reasoning.
159
160Standard API rates per million tokens are $10 input, $50 output, $1 cache reads,
161and $12.50 cache writes. Above **272,000 total input tokens**, including cache
162reads and writes, rates for the entire request become $20 input, $75 output,
163$2 cache reads, and $25 cache writes. The 500k working budget does not avoid
164this pricing tier. Codex catalog dollar estimates use these API-equivalent
165rates, not subscription billing or quota measurements.
166
167Sources: [Astra model specifications](https://developers.openai.com/api/docs/models/gpt-6-astra)
168and [prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching).
169
170## Session identity and caching
171
172zut supplies every provider request with a root cache identity, a
173conversation-thread identity, and a per-prompt turn ID. They are opaque local
174correlations; adapters translate them only when their exact provider, model,
175and endpoint declaration supports it. Resident children share the root cache
176identity but retain distinct thread identities. The OpenAI/Codex Responses
177route uses the root identity as `prompt_cache_key`; declared Codex routes also
178receive their documented session and thread routing headers. A generic
179OpenAI-compatible endpoint never receives these extensions merely because it
180accepts an OpenAI-shaped request. JSON event mode also reports sanitized cache
181diagnostics (`eligible`, `mode`, `transport`, and `continuation`). `eligible`
182is a conservative local prompt-length estimate; OpenAI performs the authoritative
183tokenization and cache decision. These records never include prompts, durable
184IDs, request bodies, or credentials.
185
186When a provider reports cache detail, zut displays cache-read hit rate as
187`cache reads / cache-reporting prompt input`. A shown `0%` is a confirmed
188cache miss; no percentage means the provider or historical session did not
189report cache detail. Cache writes remain separate from that hit rate. Provider
190retention and eviction, request routing, and OpenAI's 1,024-token eligibility
191minimum can still produce a miss even when the stable prefix is unchanged.
192
193On the official public OpenAI Responses endpoint, GPT-5.6 and GPT-6 Astra
194requests include an explicit cache breakpoint after the stable instruction/tool
195prefix while retaining implicit caching for conversation history. The Codex
196subscription route and custom Responses endpoints do not receive these explicit
197cache controls. Changing request-level reasoning effort can still disrupt prefix
198reuse: zut does not yet send cache-preserving `configuration_update` items.
199
200### OpenAI Responses WebSocket mode
201
202For the exact public endpoint `https://api.openai.com/v1/responses`, zut uses
203the Responses API WebSocket mode for the OpenAI Responses route. It keeps one
204connection per conversation thread, sends one `response.create` at a time,
205and continues the most recent completed response with `previous_response_id`
206only when the complete response-context configuration and prior input prefix
207match exactly, including the completed assistant output that precedes a new
208suffix. Compatible continuations send only new tool or user input.
209The request uses `store:false`; its only provider-specific handshake header is
210`Authorization: Bearer ...`, never the
211ChatGPT/Codex account, routing, or session headers.
212
213The capability is deliberately not inherited by `--base-url`, compatible
214providers, Azure, or the ChatGPT Codex subscription endpoint. Those clients
215continue to use their declared HTTP/SSE contracts. A request without a logical
216session ID, a failed WebSocket setup,
217an injected nonstandard RoundTripper, or a configured proxy falls back to
218HTTP/SSE with the same logical session/cache identity. Cancellation closes
219only the active session socket; a normally completed turn keeps it warm for
220the next continuation.
221
222
223## Local llama.cpp router
224
225The `llama.cpp` provider connects to a multi-model router and is separate from the `ollama` provider. Ollama normally uses `http://localhost:11434`; entering that URL for llama.cpp management produces a 404 because Ollama does not implement the router endpoints.
226
227Install a current llama.cpp build. On macOS:
228
229```bash
230brew install llama.cpp
231# use `brew upgrade llama.cpp` for an existing installation
232```
233
234Start `llama-server` without `--model`, `-m`, or `-hf`. Those flags select one model and disable the router behavior zut expects.
235
236```bash
237mkdir -p ~/llama-models
238
239llama-server \
240  --models-dir ~/llama-models \
241  --no-models-autoload \
242  --jinja \
243  --host 127.0.0.1 \
244  --port 8080 \
245  -ngl 999 \
246  -c 32768
247```
248
249This configuration discovers GGUF files below `~/llama-models`, leaves model loading under explicit user control, enables chat templates, requests GPU offload, and sets a 32K context. Tune the GPU layers and context size for the available memory.
250
251Check the management API before opening zut:
252
253```bash
254curl http://127.0.0.1:8080/health
255curl http://127.0.0.1:8080/models
256```
257
258`/models` must return JSON with a `data` array. A 404 indicates the wrong URL, an older server build, or single-model mode.
259
260To save the connection, run `/login`, choose `api key`, and select `llama.cpp`. Enter `http://127.0.0.1:8080`, without `/v1`, followed by an optional bearer key. zut validates the URL and stores both values in `$ZUT_HOME/auth.json`. Environment variables override the saved connection:
261
262```bash
263export LLAMA_BASE_URL=http://127.0.0.1:8080
264export LLAMA_API_KEY=optional-secret
265```
266
267If a key is configured, start the router with the same `--api-key` value. A server bound only to `127.0.0.1` normally does not need authentication.
268
269After login, `/llama` becomes visible. It shows model state, searches Hugging Face for GGUF repositories, offers available quantizations, reports download and load progress, and explicitly loads or unloads models. Press `d` on a router-downloaded cache model to ask the router to remove it after confirmation. Models discovered through `--models-dir` or a preset are not removable through the router API and must be deleted from their configured source. Some llama.cpp versions retain shared Hugging Face repository artifacts such as `mmproj` files after removing the selected GGUF; remove that repository's cache directory manually if those artifacts are no longer needed. `HF_TOKEN` raises Hugging Face API limits and may be required for repository metadata. For gated files, approve access first and provide the authorized token to the `llama-server` process because the router performs the download.
270
271Opening `/model` refreshes the router and displays its loaded models under provider `llama.cpp`. Unloaded models cannot handle inference and therefore remain exclusive to `/llama` until loaded. Inference is sent to the router's derived `/v1` endpoint.
272
273Ollama downloads are stored in Ollama's internal layout and do not automatically become llama.cpp models. Obtain a GGUF copy through `/llama` or place GGUF files under `~/llama-models`, then restart the router after adding files manually.
274
275## Cloud providers
276
277### Amazon Bedrock
278
279Bedrock is configured with AWS credentials, not a generic zut API-key entry.
280Use one of these credential sources:
281
282```bash
283# AWS profile
284export AWS_PROFILE=your-profile
285
286# IAM access keys
287export AWS_ACCESS_KEY_ID=AKIA...
288export AWS_SECRET_ACCESS_KEY=...
289export AWS_SESSION_TOKEN=... # only for temporary credentials
290
291# Bedrock API key bearer token
292export AWS_BEARER_TOKEN_BEDROCK=bedrock-api-key-...
293
294# Region
295export AWS_REGION=us-east-1
296```
297
298ECS task roles, IRSA, and other AWS SDK credential-chain sources are also
299supported.
300
301Example:
302
303```bash
304AWS_BEARER_TOKEN_BEDROCK=bedrock-api-key-... AWS_REGION=us-east-1 \
305  zut --provider amazon-bedrock --model anthropic.claude-sonnet-4-5-20250929-v1:0
306```
307
308Some Bedrock models require regional inference-profile IDs for on-demand
309throughput, such as `us.` or `eu.` prefixed model IDs. zut rewrites known
310families automatically where possible. Claude Opus 5 uses
311`global.anthropic.claude-opus-5`; zut maps the bare foundation-model ID to that
312global profile and supports its adaptive reasoning levels. Explicit profile IDs
313and ARNs are left unchanged.
314
315### Google Vertex AI
316
317Vertex can use a Google API key when available:
318
319```bash
320export GOOGLE_CLOUD_API_KEY=...
321zut --provider google-vertex
322```
323
324For service-account or application-default credentials, set the standard
325Google environment variables used by your deployment.
326
327### Cloudflare AI Gateway
328
329Cloudflare AI Gateway needs a Cloudflare token plus account and gateway IDs:
330
331```bash
332export CLOUDFLARE_API_KEY=...
333export CLOUDFLARE_ACCOUNT_ID=...
334export CLOUDFLARE_GATEWAY_ID=...
335zut --provider cloudflare-ai-gateway
336```
337
338### Cloudflare Workers AI
339
340Workers AI needs a Cloudflare token and account ID:
341
342```bash
343export CLOUDFLARE_API_KEY=...
344export CLOUDFLARE_ACCOUNT_ID=...
345zut --provider cloudflare-workers-ai
346```
347
348### Azure OpenAI Responses
349
350```bash
351export AZURE_OPENAI_API_KEY=...
352export AZURE_OPENAI_BASE_URL=https://your-resource.openai.azure.com
353export AZURE_OPENAI_API_VERSION=v1 # optional, v1 is the default
354zut --provider azure-openai-responses
355```
356
357The provider uses Azure's Responses API. If deployment names differ from zut
358model IDs, map them without changing the catalog:
359
360```bash
361export AZURE_OPENAI_DEPLOYMENT_NAME_MAP='gpt-5.6-luna=luna-preview,gpt-5.6-sol=sol-preview'
362```
363
364## Auth file
365
366Credentials are stored in `$ZUT_HOME/auth.json` with user-only permissions
367when zut creates the file.
368
369Example:
370
371```json
372{
373  "anthropic": { "api_key": "sk-ant-..." },
374  "openai": { "api_key": "sk-..." },
375  "google": { "api_key": "..." },
376  "additional_api_key_creds": {
377    "openrouter": { "api_key": "..." },
378    "mistral": { "api_key": "..." }
379  }
380}
381```
382
383The top-level keys are used for providers with dedicated credential fields.
384Other API-key providers are stored under `additional_api_key_creds`. Prefer
385`/login` so zut writes the correct schema.
386
387## Custom providers and models
388
389Use `$ZUT_HOME/models.json` for private models, deployment aliases, local
390servers, or OpenAI-compatible gateways that are not in the built-in catalog.
391User entries override built-in entries with the same provider and model ID, and
392adding a `models.json` no longer hides the built-in catalog: your entries are
393merged on top of the baked-in and live-discovered models.
394
395A top-level provider key that is not a built-in id defines a custom provider.
396Give it a provider-level `baseUrl` and an `api` wire format (`openai` for
397OpenAI-compatible Chat Completions, the default, `openai-responses` for the
398OpenAI Responses API, or `anthropic` for the Anthropic Messages API). A
399model-level `baseUrl` overrides the provider-level one for that model, and an
400unknown `api` value falls back to `openai` with a warning.
401
402```json
403{
404  "providers": {
405    "my-company": {
406      "baseUrl": "https://llm.mycompany.com/v1",
407      "api": "openai",
408      "models": [
409        { "id": "company-llm-v2", "name": "Company LLM v2" }
410      ]
411    }
412  }
413}
414```
415
416Custom providers are first-class: they appear in `--list-models`, `/model`, and
417`/login`. `models.json` never stores secrets. Supply the key through `/login`,
418`--api-key`, or a derived environment variable in upper snake case, so
419`my-company` reads `MY_COMPANY_API_KEY`. Because many self-hosted gateways do
420not expose a model-list endpoint, custom provider keys are accepted and stored
421without a verification probe; an invalid key surfaces on the first model call.
422
423To retrieve this custom provider's key from a password manager, add a matching
424entry to `$ZUT_HOME/auth.json`:
425
426```json
427{
428  "additional_api_key_creds": {
429    "my-company": {
430      "api_key_command": {
431        "program": "op",
432        "args": ["read", "op://Work/OpenAI/credential"],
433        "timeout_ms": 120000
434      }
435    }
436  }
437}
438```
439
440The provider IDs in `models.json` and `auth.json` must match. Then select the
441custom model directly:
442
443```sh
444zut --provider my-company --model company-llm-v2
445```
446
447## Credential resolution
448
449For each request, zut checks credentials in this order:
450
4511. Explicit CLI key, such as `--api-key`.
4522. Provider-specific environment variables (including derived custom-provider
453   variables such as `MY_COMPANY_API_KEY`).
4543. `$ZUT_HOME/auth.json`, including custom provider keys saved by `/login`.
455
456`models.json` itself never stores credentials; it only describes models and
457their endpoints.
458
459Bedrock then uses the AWS SDK credential chain for the actual request.

providers.md RawHistoryFolder

zut providers

HTTP proxy

Login methods

Command-backed API keys

Subscription providers

API-key providers

Fast mode

GPT-6 Astra

Session identity and caching

OpenAI Responses WebSocket mode

Local llama.cpp router

Cloud providers

Amazon Bedrock

Google Vertex AI

Cloudflare AI Gateway

Cloudflare Workers AI

Azure OpenAI Responses

Auth file

Custom providers and models

Credential resolution

providers.md Raw History Folder