troubleshooting.md Raw History Folder

Troubleshooting

Common issues and solutions when using Gordon.

Registry Issues

"unauthorized: authentication required"

Cause: Registry authentication is enabled but credentials are missing or invalid.

Solutions:

Login to registry:
```
docker login registry.mydomain.com
```
Check token is valid:
```
gordon auth token list
```

Generate new token:

gordon auth token generate --subject myuser --expiry 0

"connection refused" on push

Cause: Gordon is not running or registry port is blocked.

Solutions:

Check Gordon is running:

systemctl --user status gordon
# or
pgrep -f "gordon serve"

Check registry port is accessible:
```
curl -v http://localhost:5000/v2/
```
Check firewall:
```
sudo firewall-cmd --list-ports
```

"server gave HTTP response to HTTPS client"

Cause: Docker/Podman defaults to HTTPS but your registry is serving plain HTTP (common with localhost:5000 or when TLS is terminated by a reverse proxy).

Solution — Docker:

Add the registry to /etc/docker/daemon.json:

{
  "insecure-registries": ["gordon.mydomain.com:5000"]
}

Then restart Docker:

sudo systemctl restart docker

Solution — Podman:

Create or edit ~/.config/containers/registries.conf.d/gordon.conf:

[[registry]]
location = "gordon.mydomain.com:5000"
insecure = true

No restart needed — Podman reads this on each pull/push.

Note: For localhost:5000, Docker already allows insecure access by default. This is only needed for remote or domain-based registry access over HTTP.

TLS certificate errors with the Gordon CLI

Cause: The Gordon server uses a self-signed certificate or TLS is terminated elsewhere.

Solutions (in priority order):

Per-command flag:
```
gordon push myapp:latest --insecure
```
Environment variable:
```
export GORDON_INSECURE=true
```

Per-remote in ~/.config/gordon/remotes.toml:

[remotes.my-server]
url = "https://gordon.mydomain.com"
token = "..."
insecure_tls = true

Global default in ~/.config/gordon/gordon.toml:
```
[client]
insecure_tls = true
```

"unknown: image not found"

Cause: The app manifest references an image that is unavailable to the runtime.

Solution: Push the image to the configured registry, verify the service's image reference in the app manifest, then apply and deploy the accepted revision:

gordon apps apply --file app.toml
gordon apps deploy app

Deployment Issues

Container starts but app not accessible

Causes and solutions:

Wrong port exposed: Check your Dockerfile exposes the correct port:
```
EXPOSE 3000
```
Multiple ports - wrong one selected: Add gordon.proxy.port label:
```
LABEL gordon.proxy.port=3000
```
App not listening on 0.0.0.0: Ensure app binds to 0.0.0.0, not 127.0.0.1:
```
app.listen(3000, '0.0.0.0');
```

Container keeps restarting

Cause: Application crashing on startup.

Solutions:

Check workload logs:
```
gordon apps logs blog --service web
```
Check Gordon logs:
```
gordon logs -f
```

Run container manually to debug:

docker run -it registry.mydomain.com/myapp:latest sh

Environment variables not loaded

Cause: Env file not found or wrong format.

Solutions:

Check file exists with correct name:

ls ~/.gordon/env/
# Should show: app_mydomain_com.env (dots → underscores)

Check file permissions:

chmod 600 ~/.gordon/env/app_mydomain_com.env

Check file format (no spaces around =):

# Correct
KEY=value

# Wrong
KEY = value

Secrets not resolved

Cause: Secret provider not available or secret not found.

Solutions:

For pass backend:

# Check pass is available
which pass

# Check secret exists
pass show myapp/db-password

For sops backend:

# Check sops is available
which sops

# Check file can be decrypted
sops -d secrets.yaml

Network Issues

Containers can't reach each other

Cause: Network isolation enabled but services not in same network.

Solutions:

Check that both services declare the same shared network in the app manifest.
Check containers are attached to that network:
```
docker network inspect NETWORK_NAME
```

Use the service name as the internal hostname:

// Correct
connect("postgresql://postgres:5432/mydb")

// Wrong
connect("postgresql://localhost:5432/mydb")

DNS resolution failing

Cause: Container can't resolve service names.

Solutions:

Verify network isolation is enabled:
```
[network_isolation]
enabled = true
```

Check both containers in same network:

docker network inspect gordon-app-mydomain-com

Volume Issues

Data not persisting

Cause: Volume not configured or not preserved.

Solutions:

Add VOLUME to Dockerfile:
```
VOLUME ["/data"]
```

Check volume preservation:

[volumes]
preserve = true  # default: true

List volumes:
```
docker volume ls | grep gordon
```

Disk full

Cause: Registry or logs consuming too much space.

Solutions:

Check disk usage:
```
du -sh ~/.gordon/*
```
Prune unused images:
```
docker image prune -a
```

Configure log rotation:

[logging.file]
max_size = 100
max_backups = 3

Configuration Issues

Config not reloading

Cause: Gordon not watching config file.

Solution: Manual reload:

gordon reload

Stale `targets.toml` in config directory

Cause: Gordon renamed targets.toml to remotes.toml in v2.9. The old file is ignored but may cause confusion.

Solution: Delete the old file and use remotes.toml:

rm ~/.config/gordon/targets.toml

If you had remotes configured in targets.toml, recreate them:

gordon remotes add my-server https://gordon.mydomain.com --token YOUR_TOKEN

"failed to read config file"

Cause: TOML syntax error.

Solution: Validate TOML:

# Check for syntax errors
cat ~/.config/gordon/gordon.toml | tomlv
# or use online validator

Logging Issues

No logs appearing

Cause: File logging not enabled.

Solution: Enable in config:

[logging.file]
enabled = true
path = "~/.gordon/logs/gordon.log"

Workload logs unavailable

gordon apps logs APP --service SERVICE reads directly from the container runtime. Confirm that the app has an active deployment, use the exact service name, and check the runtime's logging driver and retention settings. Gordon does not write workload logs to logging.container_logs files.

Diagnostic Commands

# Check Gordon status
systemctl --user status gordon

# View Gordon logs
gordon logs -f
journalctl --user -u gordon -f

# List containers
docker ps -f "label=gordon.managed=true"

# List networks
docker network ls | grep gordon

# List volumes
docker volume ls | grep gordon

# Check workload logs through Gordon
gordon apps logs blog --service web

# Inspect runtime containers when diagnosing locally
docker ps -f "label=gordon.app=blog"

# Check connectivity
curl -v http://localhost:5000/v2/
curl -v http://localhost:8088/

Getting Help

source

1# Troubleshooting
2
3Common issues and solutions when using Gordon.
4
5## Registry Issues
6
7### "unauthorized: authentication required"
8
9**Cause:** Registry authentication is enabled but credentials are missing or invalid.
10
11**Solutions:**
12
131. Login to registry:
14   ```bash
15   docker login registry.mydomain.com
16   ```
17
182. Check token is valid:
19   ```bash
20   gordon auth token list
21   ```
22
233. Generate new token:
24   ```bash
25   gordon auth token generate --subject myuser --expiry 0
26   ```
27
28### "connection refused" on push
29
30**Cause:** Gordon is not running or registry port is blocked.
31
32**Solutions:**
33
341. Check Gordon is running:
35   ```bash
36   systemctl --user status gordon
37   # or
38   pgrep -f "gordon serve"
39   ```
40
412. Check registry port is accessible:
42   ```bash
43   curl -v http://localhost:5000/v2/
44   ```
45
463. Check firewall:
47   ```bash
48   sudo firewall-cmd --list-ports
49   ```
50
51### "server gave HTTP response to HTTPS client"
52
53**Cause:** Docker/Podman defaults to HTTPS but your registry is serving plain HTTP (common with `localhost:5000` or when TLS is terminated by a reverse proxy).
54
55**Solution — Docker:**
56
57Add the registry to `/etc/docker/daemon.json`:
58
59```json
60{
61  "insecure-registries": ["gordon.mydomain.com:5000"]
62}
63```
64
65Then restart Docker:
66
67```bash
68sudo systemctl restart docker
69```
70
71**Solution — Podman:**
72
73Create or edit `~/.config/containers/registries.conf.d/gordon.conf`:
74
75```toml
76[[registry]]
77location = "gordon.mydomain.com:5000"
78insecure = true
79```
80
81No restart needed — Podman reads this on each pull/push.
82
83> **Note:** For `localhost:5000`, Docker already allows insecure access by default. This is only needed for remote or domain-based registry access over HTTP.
84
85### TLS certificate errors with the Gordon CLI
86
87**Cause:** The Gordon server uses a self-signed certificate or TLS is terminated elsewhere.
88
89**Solutions (in priority order):**
90
911. Per-command flag:
92
93   ```bash
94   gordon push myapp:latest --insecure
95   ```
96
972. Environment variable:
98
99   ```bash
100   export GORDON_INSECURE=true
101   ```
102
1033. Per-remote in `~/.config/gordon/remotes.toml`:
104
105   ```toml
106   [remotes.my-server]
107   url = "https://gordon.mydomain.com"
108   token = "..."
109   insecure_tls = true
110   ```
111
1124. Global default in `~/.config/gordon/gordon.toml`:
113
114   ```toml
115   [client]
116   insecure_tls = true
117   ```
118
119### "unknown: image not found"
120
121**Cause:** The app manifest references an image that is unavailable to the runtime.
122
123**Solution:** Push the image to the configured registry, verify the service's `image` reference in the app manifest, then apply and deploy the accepted revision:
124
125```bash
126gordon apps apply --file app.toml
127gordon apps deploy app
128```
129
130## Deployment Issues
131
132### Container starts but app not accessible
133
134**Causes and solutions:**
135
1361. **Wrong port exposed:**
137   Check your Dockerfile exposes the correct port:
138   ```dockerfile
139   EXPOSE 3000
140   ```
141
1422. **Multiple ports - wrong one selected:**
143   Add `gordon.proxy.port` label:
144   ```dockerfile
145   LABEL gordon.proxy.port=3000
146   ```
147
1483. **App not listening on 0.0.0.0:**
149   Ensure app binds to `0.0.0.0`, not `127.0.0.1`:
150   ```javascript
151   app.listen(3000, '0.0.0.0');
152   ```
153
154### Container keeps restarting
155
156**Cause:** Application crashing on startup.
157
158**Solutions:**
159
1601. Check workload logs:
161   ```bash
162   gordon apps logs blog --service web
163   ```
164
1652. Check Gordon logs:
166   ```bash
167   gordon logs -f
168   ```
169
1703. Run container manually to debug:
171   ```bash
172   docker run -it registry.mydomain.com/myapp:latest sh
173   ```
174
175### Environment variables not loaded
176
177**Cause:** Env file not found or wrong format.
178
179**Solutions:**
180
1811. Check file exists with correct name:
182   ```bash
183   ls ~/.gordon/env/
184   # Should show: app_mydomain_com.env (dots → underscores)
185   ```
186
1872. Check file permissions:
188   ```bash
189   chmod 600 ~/.gordon/env/app_mydomain_com.env
190   ```
191
1923. Check file format (no spaces around `=`):
193   ```bash
194   # Correct
195   KEY=value
196
197   # Wrong
198   KEY = value
199   ```
200
201### Secrets not resolved
202
203**Cause:** Secret provider not available or secret not found.
204
205**Solutions:**
206
2071. For `pass` backend:
208   ```bash
209   # Check pass is available
210   which pass
211
212   # Check secret exists
213   pass show myapp/db-password
214   ```
215
2162. For `sops` backend:
217   ```bash
218   # Check sops is available
219   which sops
220
221   # Check file can be decrypted
222   sops -d secrets.yaml
223   ```
224
225## Network Issues
226
227### Containers can't reach each other
228
229**Cause:** Network isolation enabled but services not in same network.
230
231**Solutions:**
232
2331. Check that both services declare the same shared network in the app manifest.
234
2352. Check containers are attached to that network:
236   ```bash
237   docker network inspect NETWORK_NAME
238   ```
239
2403. Use the service name as the internal hostname:
241   ```javascript
242   // Correct
243   connect("postgresql://postgres:5432/mydb")
244
245   // Wrong
246   connect("postgresql://localhost:5432/mydb")
247   ```
248
249### DNS resolution failing
250
251**Cause:** Container can't resolve service names.
252
253**Solutions:**
254
2551. Verify network isolation is enabled:
256   ```toml
257   [network_isolation]
258   enabled = true
259   ```
260
2612. Check both containers in same network:
262   ```bash
263   docker network inspect gordon-app-mydomain-com
264   ```
265
266## Volume Issues
267
268### Data not persisting
269
270**Cause:** Volume not configured or not preserved.
271
272**Solutions:**
273
2741. Add VOLUME to Dockerfile:
275   ```dockerfile
276   VOLUME ["/data"]
277   ```
278
2792. Check volume preservation:
280   ```toml
281   [volumes]
282   preserve = true  # default: true
283   ```
284
2853. List volumes:
286   ```bash
287   docker volume ls | grep gordon
288   ```
289
290### Disk full
291
292**Cause:** Registry or logs consuming too much space.
293
294**Solutions:**
295
2961. Check disk usage:
297   ```bash
298   du -sh ~/.gordon/*
299   ```
300
3012. Prune unused images:
302   ```bash
303   docker image prune -a
304   ```
305
3063. Configure log rotation:
307   ```toml
308   [logging.file]
309   max_size = 100
310   max_backups = 3
311   ```
312
313## Configuration Issues
314
315### Config not reloading
316
317**Cause:** Gordon not watching config file.
318
319**Solution:** Manual reload:
320```bash
321gordon reload
322```
323
324### Stale `targets.toml` in config directory
325
326**Cause:** Gordon renamed `targets.toml` to `remotes.toml` in v2.9. The old file is ignored but may cause confusion.
327
328**Solution:** Delete the old file and use `remotes.toml`:
329
330```bash
331rm ~/.config/gordon/targets.toml
332```
333
334If you had remotes configured in `targets.toml`, recreate them:
335
336```bash
337gordon remotes add my-server https://gordon.mydomain.com --token YOUR_TOKEN
338```
339
340### "failed to read config file"
341
342**Cause:** TOML syntax error.
343
344**Solution:** Validate TOML:
345```bash
346# Check for syntax errors
347cat ~/.config/gordon/gordon.toml | tomlv
348# or use online validator
349```
350
351## Logging Issues
352
353### No logs appearing
354
355**Cause:** File logging not enabled.
356
357**Solution:** Enable in config:
358```toml
359[logging.file]
360enabled = true
361path = "~/.gordon/logs/gordon.log"
362```
363
364### Workload logs unavailable
365
366`gordon apps logs APP --service SERVICE` reads directly from the container runtime. Confirm that the app has an active deployment, use the exact service name, and check the runtime's logging driver and retention settings. Gordon does not write workload logs to `logging.container_logs` files.
367
368## Diagnostic Commands
369
370```bash
371# Check Gordon status
372systemctl --user status gordon
373
374# View Gordon logs
375gordon logs -f
376journalctl --user -u gordon -f
377
378# List containers
379docker ps -f "label=gordon.managed=true"
380
381# List networks
382docker network ls | grep gordon
383
384# List volumes
385docker volume ls | grep gordon
386
387# Check workload logs through Gordon
388gordon apps logs blog --service web
389
390# Inspect runtime containers when diagnosing locally
391docker ps -f "label=gordon.app=blog"
392
393# Check connectivity
394curl -v http://localhost:5000/v2/
395curl -v http://localhost:8088/
396```
397
398## Getting Help
399
400- [GitHub Issues](https://github.com/bnema/gordon/issues)
401- [GitHub Discussions](https://github.com/bnema/gordon/discussions)
402
403## Related
404
405- [Installation](../installation.md)
406- [Configuration](../config/index.md)
407- [CLI Commands](../cli/index.md)

troubleshooting.md RawHistoryFolder

Troubleshooting

Registry Issues

"unauthorized: authentication required"

"connection refused" on push

"server gave HTTP response to HTTPS client"

TLS certificate errors with the Gordon CLI

"unknown: image not found"

Deployment Issues

Container starts but app not accessible

Container keeps restarting

Environment variables not loaded

Secrets not resolved

Network Issues

Containers can't reach each other

DNS resolution failing

Volume Issues

Data not persisting

Disk full

Configuration Issues

Config not reloading

Stale targets.toml in config directory

"failed to read config file"

Logging Issues

No logs appearing

Workload logs unavailable

Diagnostic Commands

Getting Help

Related

troubleshooting.md Raw History Folder

Stale `targets.toml` in config directory