troubleshooting.md Raw History Folder

Troubleshooting

Common issues and solutions when using Gordon.

Registry Issues

"unauthorized: authentication required"

Cause: Registry authentication is enabled but credentials are missing or invalid.

Solutions:

Login to registry:
```
docker login registry.mydomain.com
```
Check token is valid:
```
gordon auth token list
```

Generate new token:

gordon auth token generate --subject myuser --expiry 0

"connection refused" on push

Cause: Gordon is not running or registry port is blocked.

Solutions:

Check Gordon is running:

systemctl --user status gordon
# or
pgrep -f "gordon serve"

Check registry port is accessible:
```
curl -v http://localhost:5000/v2/
```
Check firewall:
```
sudo firewall-cmd --list-ports
```

"server gave HTTP response to HTTPS client"

Cause: Docker/Podman defaults to HTTPS but your registry is serving plain HTTP (common with localhost:5000 or when TLS is terminated by a reverse proxy).

Solution — Docker:

Add the registry to /etc/docker/daemon.json:

{
  "insecure-registries": ["gordon.mydomain.com:5000"]
}

Then restart Docker:

sudo systemctl restart docker

Solution — Podman:

Create or edit ~/.config/containers/registries.conf.d/gordon.conf:

[[registry]]
location = "gordon.mydomain.com:5000"
insecure = true

No restart needed — Podman reads this on each pull/push.

Note: For localhost:5000, Docker already allows insecure access by default. This is only needed for remote or domain-based registry access over HTTP.

TLS certificate errors with the Gordon CLI

Cause: The Gordon server uses a self-signed certificate or TLS is terminated elsewhere.

Solutions (in priority order):

Per-command flag:

gordon images push myapp:latest --insecure

Environment variable:
```
export GORDON_INSECURE=true
```

Per-remote in ~/.config/gordon/remotes.toml:

[remotes.my-server]
url = "https://gordon.mydomain.com"
token = "..."
insecure_tls = true

Global default in ~/.config/gordon/gordon.toml:
```
[client]
insecure_tls = true
```

"unknown: image not found"

Cause: The app manifest references an image that is unavailable to the runtime.

Solution: Push the image to the configured registry, verify the service's image reference in the app manifest, then apply and deploy the accepted revision:

gordon apps apply --file app.toml
gordon apps deploy app

Deployment Issues

Container starts but app not accessible

Causes and solutions:

Wrong port exposed: Check your Dockerfile exposes the correct port:
```
EXPOSE 3000
```
Multiple ports - wrong one selected: Add gordon.proxy.port label:
```
LABEL gordon.proxy.port=3000
```
App not listening on 0.0.0.0: Ensure app binds to 0.0.0.0, not 127.0.0.1:
```
app.listen(3000, '0.0.0.0');
```

Container keeps restarting

Cause: Application crashing on startup.

Solutions:

Check workload logs:
```
gordon apps logs blog --service web
```
Check Gordon logs:
```
gordon daemon logs -f
```

Run container manually to debug:

docker run -it registry.mydomain.com/myapp:latest sh

Environment variables not loaded

Cause: The variable is not declared in the app file, or its secret value was never set.

Solutions:

Declare app-wide public values under [env], per-service public values under [services.<name>.env], and sensitive values under [services.<name>.secrets] in the app file, then apply it:
```
gordon apps apply --file ./blog.toml
```
Check which secret values are set (names only):
```
gordon apps secrets list blog
```
Deploy or restart: values apply on the next deploy/restart, never to running containers.
```
gordon apps restart blog
```

Secrets not resolved

Cause: Secret provider not available or secret not found.

Solutions:

For pass backend:

# Check pass is available
which pass

# Check secret exists
pass show myapp/db-password

For sops backend:

# Check sops is available
which sops

# Check file can be decrypted
sops -d secrets.yaml

Network Issues

Containers can't reach each other

Cause: Network isolation enabled but services not in same network.

Solutions:

Check that both services declare the same shared network in the app manifest.
Check containers are attached to that network:
```
docker network inspect NETWORK_NAME
```

Use the service name as the internal hostname:

// Correct
connect("postgresql://postgres:5432/mydb")

// Wrong
connect("postgresql://localhost:5432/mydb")

DNS resolution failing

Cause: Container can't resolve service names.

Solutions:

Verify network isolation is enabled:
```
[network_isolation]
enabled = true
```

Check both containers in same network:

docker network inspect gordon-app-mydomain-com

Volume Issues

Data not persisting

Cause: Volume not configured or not preserved.

Solutions:

Add VOLUME to Dockerfile:
```
VOLUME ["/data"]
```

Check volume preservation:

[volumes]
preserve = true  # default: true

List volumes:
```
docker volume ls | grep gordon
```

Disk full

Cause: Registry or logs consuming too much space.

Solutions:

Check disk usage:
```
du -sh ~/.gordon/*
```
Prune unused images:
```
docker image prune -a
```

Configure log rotation:

[logging.file]
max_size = 100
max_backups = 3

Configuration Issues

Config not reloading

Cause: Gordon not watching config file.

Solution: Manual reload:

gordon daemon reload

Stale `targets.toml` in config directory

Cause: Gordon renamed targets.toml to remotes.toml in v2.9. The old file is ignored but may cause confusion.

Solution: Delete the old file and use remotes.toml:

rm ~/.config/gordon/targets.toml

If you had remotes configured in targets.toml, recreate them:

gordon remotes add my-server https://gordon.mydomain.com --token YOUR_TOKEN

"failed to read config file"

Cause: TOML syntax error.

Solution: Validate TOML:

# Check for syntax errors
cat ~/.config/gordon/gordon.toml | tomlv
# or use online validator

Logging Issues

No logs appearing

Cause: File logging not enabled.

Solution: Enable in config:

[logging.file]
enabled = true
path = "~/.gordon/logs/gordon.log"

Workload logs unavailable

gordon apps logs APP --service SERVICE reads directly from the container runtime. Confirm that the app has an active deployment, use the exact service name, and check the runtime's logging driver and retention settings. Gordon does not write workload logs to logging.container_logs files.

Diagnostic Commands

# Check Gordon status
systemctl --user status gordon

# View Gordon logs
gordon daemon logs -f
journalctl --user -u gordon -f

# List containers
docker ps -f "label=gordon.managed=true"

# List networks
docker network ls | grep gordon

# List volumes
docker volume ls | grep gordon

# Check workload logs through Gordon
gordon apps logs blog --service web

# Inspect runtime containers when diagnosing locally
docker ps -f "label=gordon.app=blog"

# Check connectivity
curl -v http://localhost:5000/v2/
curl -v http://localhost:8088/

Getting Help

source

1# Troubleshooting
2
3Common issues and solutions when using Gordon.
4
5## Registry Issues
6
7### "unauthorized: authentication required"
8
9**Cause:** Registry authentication is enabled but credentials are missing or invalid.
10
11**Solutions:**
12
131. Login to registry:
14   ```bash
15   docker login registry.mydomain.com
16   ```
17
182. Check token is valid:
19   ```bash
20   gordon auth token list
21   ```
22
233. Generate new token:
24   ```bash
25   gordon auth token generate --subject myuser --expiry 0
26   ```
27
28### "connection refused" on push
29
30**Cause:** Gordon is not running or registry port is blocked.
31
32**Solutions:**
33
341. Check Gordon is running:
35   ```bash
36   systemctl --user status gordon
37   # or
38   pgrep -f "gordon serve"
39   ```
40
412. Check registry port is accessible:
42   ```bash
43   curl -v http://localhost:5000/v2/
44   ```
45
463. Check firewall:
47   ```bash
48   sudo firewall-cmd --list-ports
49   ```
50
51### "server gave HTTP response to HTTPS client"
52
53**Cause:** Docker/Podman defaults to HTTPS but your registry is serving plain HTTP (common with `localhost:5000` or when TLS is terminated by a reverse proxy).
54
55**Solution — Docker:**
56
57Add the registry to `/etc/docker/daemon.json`:
58
59```json
60{
61  "insecure-registries": ["gordon.mydomain.com:5000"]
62}
63```
64
65Then restart Docker:
66
67```bash
68sudo systemctl restart docker
69```
70
71**Solution — Podman:**
72
73Create or edit `~/.config/containers/registries.conf.d/gordon.conf`:
74
75```toml
76[[registry]]
77location = "gordon.mydomain.com:5000"
78insecure = true
79```
80
81No restart needed — Podman reads this on each pull/push.
82
83> **Note:** For `localhost:5000`, Docker already allows insecure access by default. This is only needed for remote or domain-based registry access over HTTP.
84
85### TLS certificate errors with the Gordon CLI
86
87**Cause:** The Gordon server uses a self-signed certificate or TLS is terminated elsewhere.
88
89**Solutions (in priority order):**
90
911. Per-command flag:
92
93   ```bash
94   gordon images push myapp:latest --insecure
95   ```
96
972. Environment variable:
98
99   ```bash
100   export GORDON_INSECURE=true
101   ```
102
1033. Per-remote in `~/.config/gordon/remotes.toml`:
104
105   ```toml
106   [remotes.my-server]
107   url = "https://gordon.mydomain.com"
108   token = "..."
109   insecure_tls = true
110   ```
111
1124. Global default in `~/.config/gordon/gordon.toml`:
113
114   ```toml
115   [client]
116   insecure_tls = true
117   ```
118
119### "unknown: image not found"
120
121**Cause:** The app manifest references an image that is unavailable to the runtime.
122
123**Solution:** Push the image to the configured registry, verify the service's `image` reference in the app manifest, then apply and deploy the accepted revision:
124
125```bash
126gordon apps apply --file app.toml
127gordon apps deploy app
128```
129
130## Deployment Issues
131
132### Container starts but app not accessible
133
134**Causes and solutions:**
135
1361. **Wrong port exposed:**
137   Check your Dockerfile exposes the correct port:
138   ```dockerfile
139   EXPOSE 3000
140   ```
141
1422. **Multiple ports - wrong one selected:**
143   Add `gordon.proxy.port` label:
144   ```dockerfile
145   LABEL gordon.proxy.port=3000
146   ```
147
1483. **App not listening on 0.0.0.0:**
149   Ensure app binds to `0.0.0.0`, not `127.0.0.1`:
150   ```javascript
151   app.listen(3000, '0.0.0.0');
152   ```
153
154### Container keeps restarting
155
156**Cause:** Application crashing on startup.
157
158**Solutions:**
159
1601. Check workload logs:
161   ```bash
162   gordon apps logs blog --service web
163   ```
164
1652. Check Gordon logs:
166   ```bash
167   gordon daemon logs -f
168   ```
169
1703. Run container manually to debug:
171   ```bash
172   docker run -it registry.mydomain.com/myapp:latest sh
173   ```
174
175### Environment variables not loaded
176
177**Cause:** The variable is not declared in the app file, or its secret value was never set.
178
179**Solutions:**
180
1811. Declare app-wide public values under `[env]`, per-service public values under `[services.<name>.env]`, and sensitive values under `[services.<name>.secrets]` in the app file, then apply it:
182   ```bash
183   gordon apps apply --file ./blog.toml
184   ```
185
1862. Check which secret values are set (names only):
187   ```bash
188   gordon apps secrets list blog
189   ```
190
1913. Deploy or restart: values apply on the next deploy/restart, never to running containers.
192   ```bash
193   gordon apps restart blog
194   ```
195
196### Secrets not resolved
197
198**Cause:** Secret provider not available or secret not found.
199
200**Solutions:**
201
2021. For `pass` backend:
203   ```bash
204   # Check pass is available
205   which pass
206
207   # Check secret exists
208   pass show myapp/db-password
209   ```
210
2112. For `sops` backend:
212   ```bash
213   # Check sops is available
214   which sops
215
216   # Check file can be decrypted
217   sops -d secrets.yaml
218   ```
219
220## Network Issues
221
222### Containers can't reach each other
223
224**Cause:** Network isolation enabled but services not in same network.
225
226**Solutions:**
227
2281. Check that both services declare the same shared network in the app manifest.
229
2302. Check containers are attached to that network:
231   ```bash
232   docker network inspect NETWORK_NAME
233   ```
234
2353. Use the service name as the internal hostname:
236   ```javascript
237   // Correct
238   connect("postgresql://postgres:5432/mydb")
239
240   // Wrong
241   connect("postgresql://localhost:5432/mydb")
242   ```
243
244### DNS resolution failing
245
246**Cause:** Container can't resolve service names.
247
248**Solutions:**
249
2501. Verify network isolation is enabled:
251   ```toml
252   [network_isolation]
253   enabled = true
254   ```
255
2562. Check both containers in same network:
257   ```bash
258   docker network inspect gordon-app-mydomain-com
259   ```
260
261## Volume Issues
262
263### Data not persisting
264
265**Cause:** Volume not configured or not preserved.
266
267**Solutions:**
268
2691. Add VOLUME to Dockerfile:
270   ```dockerfile
271   VOLUME ["/data"]
272   ```
273
2742. Check volume preservation:
275   ```toml
276   [volumes]
277   preserve = true  # default: true
278   ```
279
2803. List volumes:
281   ```bash
282   docker volume ls | grep gordon
283   ```
284
285### Disk full
286
287**Cause:** Registry or logs consuming too much space.
288
289**Solutions:**
290
2911. Check disk usage:
292   ```bash
293   du -sh ~/.gordon/*
294   ```
295
2962. Prune unused images:
297   ```bash
298   docker image prune -a
299   ```
300
3013. Configure log rotation:
302   ```toml
303   [logging.file]
304   max_size = 100
305   max_backups = 3
306   ```
307
308## Configuration Issues
309
310### Config not reloading
311
312**Cause:** Gordon not watching config file.
313
314**Solution:** Manual reload:
315```bash
316gordon daemon reload
317```
318
319### Stale `targets.toml` in config directory
320
321**Cause:** Gordon renamed `targets.toml` to `remotes.toml` in v2.9. The old file is ignored but may cause confusion.
322
323**Solution:** Delete the old file and use `remotes.toml`:
324
325```bash
326rm ~/.config/gordon/targets.toml
327```
328
329If you had remotes configured in `targets.toml`, recreate them:
330
331```bash
332gordon remotes add my-server https://gordon.mydomain.com --token YOUR_TOKEN
333```
334
335### "failed to read config file"
336
337**Cause:** TOML syntax error.
338
339**Solution:** Validate TOML:
340```bash
341# Check for syntax errors
342cat ~/.config/gordon/gordon.toml | tomlv
343# or use online validator
344```
345
346## Logging Issues
347
348### No logs appearing
349
350**Cause:** File logging not enabled.
351
352**Solution:** Enable in config:
353```toml
354[logging.file]
355enabled = true
356path = "~/.gordon/logs/gordon.log"
357```
358
359### Workload logs unavailable
360
361`gordon apps logs APP --service SERVICE` reads directly from the container runtime. Confirm that the app has an active deployment, use the exact service name, and check the runtime's logging driver and retention settings. Gordon does not write workload logs to `logging.container_logs` files.
362
363## Diagnostic Commands
364
365```bash
366# Check Gordon status
367systemctl --user status gordon
368
369# View Gordon logs
370gordon daemon logs -f
371journalctl --user -u gordon -f
372
373# List containers
374docker ps -f "label=gordon.managed=true"
375
376# List networks
377docker network ls | grep gordon
378
379# List volumes
380docker volume ls | grep gordon
381
382# Check workload logs through Gordon
383gordon apps logs blog --service web
384
385# Inspect runtime containers when diagnosing locally
386docker ps -f "label=gordon.app=blog"
387
388# Check connectivity
389curl -v http://localhost:5000/v2/
390curl -v http://localhost:8088/
391```
392
393## Getting Help
394
395- [GitHub Issues](https://github.com/bnema/gordon/issues)
396- [GitHub Discussions](https://github.com/bnema/gordon/discussions)
397
398## Related
399
400- [Installation](../installation.md)
401- [Configuration](../config/index.md)
402- [CLI Commands](../cli/index.md)

troubleshooting.md RawHistoryFolder

Troubleshooting

Registry Issues

"unauthorized: authentication required"

"connection refused" on push

"server gave HTTP response to HTTPS client"

TLS certificate errors with the Gordon CLI

"unknown: image not found"

Deployment Issues

Container starts but app not accessible

Container keeps restarting

Environment variables not loaded

Secrets not resolved

Network Issues

Containers can't reach each other

DNS resolution failing

Volume Issues

Data not persisting

Disk full

Configuration Issues

Config not reloading

Stale targets.toml in config directory

"failed to read config file"

Logging Issues

No logs appearing

Workload logs unavailable

Diagnostic Commands

Getting Help

Related

troubleshooting.md Raw History Folder

Stale `targets.toml` in config directory