1# Health Monitoring with systemd
2
3Monitor your Gordon routes and get desktop notifications when something goes down. Uses `gordon routes list --json`, a bash script, and a systemd user timer.
4
5## Prerequisites
6
7- Gordon CLI installed
8- `jq` and `notify-send` (libnotify) available
9- systemd user session (most Linux desktops)
10
11For remote instances, configure a saved remote with `gordon remotes add` and `gordon remotes use` (see [Remote CLI Management](./remote-cli.md)). The CLI uses the active remote automatically.
12
13---
14
15## The Check Script
16
17Create `~/.local/bin/gordon-watchdog.sh`:
18
19```bash
20#!/usr/bin/env bash
21set -euo pipefail
22
23GORDON_CMD="gordon routes list --json"
24# For remote monitoring, configure a saved remote first:
25#   gordon remotes add myserver https://gordon.example.com --token-env GORDON_TOKEN
26#   gordon remotes use myserver
27# The CLI then targets the remote automatically.
28
29json=$($GORDON_CMD) || {
30    notify-send -u critical "Gordon Watchdog" "Cannot reach Gordon"
31    exit 1
32}
33
34problems=()
35
36while IFS= read -r route; do
37    domain=$(echo "$route" | jq -r '.domain')
38    http_status=$(echo "$route" | jq -r '.http_status')
39    container_status=$(echo "$route" | jq -r '.container_status')
40
41    # HTTP status: flag 4xx, 5xx, 0 (no response), or unavailable
42    if ! [[ "$http_status" =~ ^[0-9]+$ ]]; then
43        problems+=("$domain: HTTP status unavailable ($http_status)")
44    elif [[ "$http_status" -eq 0 ]] || [[ "$http_status" -ge 400 ]]; then
45        problems+=("$domain: HTTP $http_status")
46    fi
47
48    # Container not running
49    if [[ "$container_status" != "running" ]]; then
50        problems+=("$domain: container $container_status")
51    fi
52
53    # Check attachments (databases, sidecars)
54    while IFS= read -r att; do
55        att_name=$(echo "$att" | jq -r '.name')
56        att_status=$(echo "$att" | jq -r '.status')
57        if [[ "$att_status" != "running" ]]; then
58            problems+=("$domain/$att_name: $att_status")
59        fi
60    done < <(echo "$route" | jq -c '.attachments[]? // empty')
61
62done < <(echo "$json" | jq -c '.[]')
63
64if [[ ${#problems[@]} -gt 0 ]]; then
65    body=$(printf '%s\n' "${problems[@]}")
66    notify-send -u critical -i dialog-warning \
67        "Gordon Watchdog" \
68        "${#problems[@]} problem(s):\n$body"
69    echo "[$(date -Iseconds)] ALERT: ${#problems[@]} problem(s)"
70    printf '  %s\n' "${problems[@]}"
71else
72    echo "[$(date -Iseconds)] OK: all routes healthy"
73fi
74```
75
76Make it executable:
77
78```bash
79chmod +x ~/.local/bin/gordon-watchdog.sh
80```
81
82Test it:
83
84```bash
85~/.local/bin/gordon-watchdog.sh
86# Expected: [YYYY-MM-DDThh:mm:ss+00:00] OK: all routes healthy
87```
88
89## What Gets Flagged
90
91| Condition | Example |
92|-----------|---------|
93| HTTP 4xx or 5xx | App returning 500 |
94| HTTP 0 | Container not responding at all |
95| Container not running | Crashed or stopped container |
96| Attachment not running | Database sidecar down |
97
98Redirects (301, 302, 303, 307) are not flagged. These are normal for apps that redirect HTTP to HTTPS or to a login page.
99
100## systemd Timer
101
102Two unit files go in `~/.config/systemd/user/`.
103
104### Service unit
105
106`gordon-watchdog.service`:
107
108```ini
109[Unit]
110Description=Gordon Routes Health Check
111After=network-online.target
112
113[Service]
114Type=oneshot
115ExecStart=%h/.local/bin/gordon-watchdog.sh
116Environment=DISPLAY=:0
117Environment=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/%U/bus
118```
119
120### Timer unit
121
122`gordon-watchdog.timer`:
123
124```ini
125[Unit]
126Description=Run Gordon Watchdog every 5 minutes
127
128[Timer]
129OnBootSec=1min
130OnUnitActiveSec=5min
131Persistent=true
132
133[Install]
134WantedBy=timers.target
135```
136
137### Enable it
138
139```bash
140systemctl --user daemon-reload
141systemctl --user enable --now gordon-watchdog.timer
142```
143
144### Verify
145
146```bash
147# Timer is active
148systemctl --user status gordon-watchdog.timer
149
150# Run it manually
151systemctl --user start gordon-watchdog.service
152
153# Check logs
154journalctl --user -u gordon-watchdog.service --since "1 hour ago"
155```
156
157---
158
159## Customization
160
161**Change the interval**: edit `OnUnitActiveSec` in the timer file. Use `1min`, `10min`, `30min`, etc.
162
163**Log to a file**: add output redirection in the service unit:
164
165```ini
166StandardOutput=append:%h/.local/share/gordon-watchdog.log
167StandardError=append:%h/.local/share/gordon-watchdog.log
168```
169
170**Send alerts elsewhere**: replace `notify-send` in the script with whatever you need. `curl` a webhook, write to a Slack channel, send an email. The `$body` variable has the list of problems, one per line.
171
172**Stricter HTTP checks**: if you want to flag anything that isn't exactly 200, replace the HTTP check with:
173
174```bash
175if ! [[ "$http_status" =~ ^[0-9]+$ ]]; then
176    problems+=("$domain: HTTP status unavailable ($http_status)")
177elif [[ "$http_status" -ne 200 ]]; then
178    problems+=("$domain: HTTP $http_status")
179fi
180```
181
182**Remote monitoring**: configure a saved remote with `gordon remotes add` and `gordon remotes use`. The script picks it up automatically from `~/.config/gordon/remotes.toml`.
183
184---
185
186## Troubleshooting
187
188### No desktop notification
189
190The systemd user service needs access to the D-Bus session bus and display. If notifications don't appear:
191
192```bash
193# Check your UID
194id -u
195# Should match the %U in the service file (usually 1000)
196
197# Verify D-Bus socket exists
198ls /run/user/$(id -u)/bus
199```
200
201On Wayland, you may also need `WAYLAND_DISPLAY` in the service environment.
202
203### "Cannot reach Gordon" on every run
204
205If Gordon runs as a user service, ensure it's started before the watchdog. Add to the service unit:
206
207```ini
208[Unit]
209After=network-online.target gordon.service
210```
211
212### Timer not firing
213
214```bash
215systemctl --user list-timers --all | grep gordon
216```
217
218If the timer shows `n/a` for next run, re-enable it:
219
220```bash
221systemctl --user reenable gordon-watchdog.timer
222systemctl --user start gordon-watchdog.timer
223```