docs(wiki): add health monitoring guide with systemd timer

Show how to use `gordon routes list --json` with a systemd user
timer to get desktop notifications when routes go down.

commit: e7e0b7ace839c55f2cfa53204463eaba34c99121
author: brice · 7mo ago · 2026-03-24 05:25 UTC
parent: 15d9fe3
signature: GPG VerifiedCD88F9E564E0102646FB6077A1A07C7C7C3E349D · arch-gpg Verified for the committer email

3 files · +222 -0

A wiki/guides/health-monitoring.md +217 -0

@@ -0,0 +1,217 @@
	1	+	# Health Monitoring with systemd
	2	+
	3	+	Monitor your Gordon routes and get desktop notifications when something goes down. Uses `gordon routes list --json`, a bash script, and a systemd user timer.
	4	+
	5	+	## Prerequisites
	6	+
	7	+	- Gordon CLI installed
	8	+	- `jq` and `notify-send` (libnotify) available
	9	+	- systemd user session (most Linux desktops)
	10	+
	11	+	For remote instances, pass `--remote` and `--token` to the gordon command (see [Remote CLI Management](./remote-cli.md)).
	12	+
	13	+	---
	14	+
	15	+	## The Check Script
	16	+
	17	+	Create `~/.local/bin/gordon-watchdog.sh`:
	18	+
	19	+	```bash
	20	+	#!/usr/bin/env bash
	21	+	set -euo pipefail
	22	+
	23	+	GORDON_CMD="gordon routes list --json"
	24	+	# For remote monitoring, use:
	25	+	# GORDON_CMD="gordon routes list --json --remote https://gordon.example.com --token $GORDON_TOKEN"
	26	+
	27	+	json=$($GORDON_CMD 2>&1) \|\| {
	28	+	notify-send -u critical "Gordon Watchdog" "Cannot reach Gordon"
	29	+	exit 1
	30	+	}
	31	+
	32	+	problems=()
	33	+
	34	+	while IFS= read -r route; do
	35	+	domain=$(echo "$route" \| jq -r '.domain')
	36	+	http_status=$(echo "$route" \| jq -r '.http_status')
	37	+	container_status=$(echo "$route" \| jq -r '.container_status')
	38	+
	39	+	# HTTP status: flag 4xx, 5xx, or 0 (no response)
	40	+	if [[ "$http_status" -eq 0 ]] \|\| [[ "$http_status" -ge 400 ]]; then
	41	+	problems+=("$domain: HTTP $http_status")
	42	+	fi
	43	+
	44	+	# Container not running
	45	+	if [[ "$container_status" != "running" ]]; then
	46	+	problems+=("$domain: container $container_status")
	47	+	fi
	48	+
	49	+	# Check attachments (databases, sidecars)
	50	+	while IFS= read -r att; do
	51	+	att_name=$(echo "$att" \| jq -r '.name')
	52	+	att_status=$(echo "$att" \| jq -r '.status')
	53	+	if [[ "$att_status" != "running" ]]; then
	54	+	problems+=("$domain/$att_name: $att_status")
	55	+	fi
	56	+	done < <(echo "$route" \| jq -c '.attachments[]? // empty')
	57	+
	58	+	done < <(echo "$json" \| jq -c '.[]')
	59	+
	60	+	if [[ ${#problems[@]} -gt 0 ]]; then
	61	+	body=$(printf '%s\n' "${problems[@]}")
	62	+	notify-send -u critical -i dialog-warning \
	63	+	"Gordon Watchdog" \
	64	+	"${#problems[@]} problem(s):\n$body"
	65	+	echo "[$(date -Iseconds)] ALERT: ${#problems[@]} problem(s)"
	66	+	printf ' %s\n' "${problems[@]}"
	67	+	else
	68	+	echo "[$(date -Iseconds)] OK: all routes healthy"
	69	+	fi
	70	+	```
	71	+
	72	+	Make it executable:
	73	+
	74	+	```bash
	75	+	chmod +x ~/.local/bin/gordon-watchdog.sh
	76	+	```
	77	+
	78	+	Test it:
	79	+
	80	+	```bash
	81	+	~/.local/bin/gordon-watchdog.sh
	82	+	# Expected: [2024-01-15T10:30:00+01:00] OK: all routes healthy
	83	+	```
	84	+
	85	+	## What Gets Flagged
	86	+
	87	+	\| Condition \| Example \|
	88	+	\|-----------\|---------\|
	89	+	\| HTTP 4xx or 5xx \| App returning 500 \|
	90	+	\| HTTP 0 \| Container not responding at all \|
	91	+	\| Container not running \| Crashed or stopped container \|
	92	+	\| Attachment not running \| Database sidecar down \|
	93	+
	94	+	Redirects (301, 302, 303, 307) are not flagged. These are normal for apps that redirect HTTP to HTTPS or to a login page.
	95	+
	96	+	## systemd Timer
	97	+
	98	+	Two unit files go in `~/.config/systemd/user/`.
	99	+
	100	+	### Service unit
	101	+
	102	+	`gordon-watchdog.service`:
	103	+
	104	+	```ini
	105	+	[Unit]
	106	+	Description=Gordon Routes Health Check
	107	+	After=network-online.target
	108	+
	109	+	[Service]
	110	+	Type=oneshot
	111	+	ExecStart=%h/.local/bin/gordon-watchdog.sh
	112	+	Environment=DISPLAY=:0
	113	+	Environment=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/%U/bus
	114	+	```
	115	+
	116	+	### Timer unit
	117	+
	118	+	`gordon-watchdog.timer`:
	119	+
	120	+	```ini
	121	+	[Unit]
	122	+	Description=Run Gordon Watchdog every 5 minutes
	123	+
	124	+	[Timer]
	125	+	OnBootSec=1min
	126	+	OnUnitActiveSec=5min
	127	+	Persistent=true
	128	+
	129	+	[Install]
	130	+	WantedBy=timers.target
	131	+	```
	132	+
	133	+	### Enable it
	134	+
	135	+	```bash
	136	+	systemctl --user daemon-reload
	137	+	systemctl --user enable --now gordon-watchdog.timer
	138	+	```
	139	+
	140	+	### Verify
	141	+
	142	+	```bash
	143	+	# Timer is active
	144	+	systemctl --user status gordon-watchdog.timer
	145	+
	146	+	# Run it manually
	147	+	systemctl --user start gordon-watchdog.service
	148	+
	149	+	# Check logs
	150	+	journalctl --user -u gordon-watchdog.service --since "1 hour ago"
	151	+	```
	152	+
	153	+	---
	154	+
	155	+	## Customization
	156	+
	157	+	Change the interval: edit `OnUnitActiveSec` in the timer file. Use `1min`, `10min`, `30min`, etc.
	158	+
	159	+	Log to a file: add output redirection in the service unit:
	160	+
	161	+	```ini
	162	+	StandardOutput=append:%h/.local/share/gordon-watchdog.log
	163	+	StandardError=append:%h/.local/share/gordon-watchdog.log
	164	+	```
	165	+
	166	+	Send alerts elsewhere: replace `notify-send` in the script with whatever you need. `curl` a webhook, write to a Slack channel, send an email. The `$body` variable has the list of problems, one per line.
	167	+
	168	+	Stricter HTTP checks: if you want to flag anything that isn't exactly 200, replace the HTTP check with:
	169	+
	170	+	```bash
	171	+	if [[ "$http_status" -ne 200 ]]; then
	172	+	problems+=("$domain: HTTP $http_status")
	173	+	fi
	174	+	```
	175	+
	176	+	Remote monitoring: set `GORDON_CMD` to include `--remote` and `--token` flags, or export `GORDON_TOKEN` as an environment variable in the service unit.
	177	+
	178	+	---
	179	+
	180	+	## Troubleshooting
	181	+
	182	+	### No desktop notification
	183	+
	184	+	The systemd user service needs access to the D-Bus session bus and display. If notifications don't appear:
	185	+
	186	+	```bash
	187	+	# Check your UID
	188	+	id -u
	189	+	# Should match the %U in the service file (usually 1000)
	190	+
	191	+	# Verify D-Bus socket exists
	192	+	ls /run/user/$(id -u)/bus
	193	+	```
	194	+
	195	+	On Wayland, you may also need `WAYLAND_DISPLAY` in the service environment.
	196	+
	197	+	### "Cannot reach Gordon" on every run
	198	+
	199	+	If Gordon runs as a user service, ensure it's started before the watchdog. Add to the service unit:
	200	+
	201	+	```ini
	202	+	[Unit]
	203	+	After=network-online.target gordon.service
	204	+	```
	205	+
	206	+	### Timer not firing
	207	+
	208	+	```bash
	209	+	systemctl --user list-timers --all \| grep gordon
	210	+	```
	211	+
	212	+	If the timer shows `n/a` for next run, re-enable it:
	213	+
	214	+	```bash
	215	+	systemctl --user reenable gordon-watchdog.timer
	216	+	systemctl --user start gordon-watchdog.timer
	217	+	```

@@ -0,0 +1,217 @@
	1	+	# Health Monitoring with systemd
	2	+
	3	+	Monitor your Gordon routes and get desktop notifications when something goes down. Uses `gordon routes list --json`, a bash script, and a systemd user timer.
	4	+
	5	+	## Prerequisites
	6	+
	7	+	- Gordon CLI installed
	8	+	- `jq` and `notify-send` (libnotify) available
	9	+	- systemd user session (most Linux desktops)
	10	+
	11	+	For remote instances, pass `--remote` and `--token` to the gordon command (see [Remote CLI Management](./remote-cli.md)).
	12	+
	13	+	---
	14	+
	15	+	## The Check Script
	16	+
	17	+	Create `~/.local/bin/gordon-watchdog.sh`:
	18	+
	19	+	```bash
	20	+	#!/usr/bin/env bash
	21	+	set -euo pipefail
	22	+
	23	+	GORDON_CMD="gordon routes list --json"
	24	+	# For remote monitoring, use:
	25	+	# GORDON_CMD="gordon routes list --json --remote https://gordon.example.com --token $GORDON_TOKEN"
	26	+
	27	+	json=$($GORDON_CMD 2>&1) \|\| {
	28	+	notify-send -u critical "Gordon Watchdog" "Cannot reach Gordon"
	29	+	exit 1
	30	+	}
	31	+
	32	+	problems=()
	33	+
	34	+	while IFS= read -r route; do
	35	+	domain=$(echo "$route" \| jq -r '.domain')
	36	+	http_status=$(echo "$route" \| jq -r '.http_status')
	37	+	container_status=$(echo "$route" \| jq -r '.container_status')
	38	+
	39	+	# HTTP status: flag 4xx, 5xx, or 0 (no response)
	40	+	if [[ "$http_status" -eq 0 ]] \|\| [[ "$http_status" -ge 400 ]]; then
	41	+	problems+=("$domain: HTTP $http_status")
	42	+	fi
	43	+
	44	+	# Container not running
	45	+	if [[ "$container_status" != "running" ]]; then
	46	+	problems+=("$domain: container $container_status")
	47	+	fi
	48	+
	49	+	# Check attachments (databases, sidecars)
	50	+	while IFS= read -r att; do
	51	+	att_name=$(echo "$att" \| jq -r '.name')
	52	+	att_status=$(echo "$att" \| jq -r '.status')
	53	+	if [[ "$att_status" != "running" ]]; then
	54	+	problems+=("$domain/$att_name: $att_status")
	55	+	fi
	56	+	done < <(echo "$route" \| jq -c '.attachments[]? // empty')
	57	+
	58	+	done < <(echo "$json" \| jq -c '.[]')
	59	+
	60	+	if [[ ${#problems[@]} -gt 0 ]]; then
	61	+	body=$(printf '%s\n' "${problems[@]}")
	62	+	notify-send -u critical -i dialog-warning \
	63	+	"Gordon Watchdog" \
	64	+	"${#problems[@]} problem(s):\n$body"
	65	+	echo "[$(date -Iseconds)] ALERT: ${#problems[@]} problem(s)"
	66	+	printf ' %s\n' "${problems[@]}"
	67	+	else
	68	+	echo "[$(date -Iseconds)] OK: all routes healthy"
	69	+	fi
	70	+	```
	71	+
	72	+	Make it executable:
	73	+
	74	+	```bash
	75	+	chmod +x ~/.local/bin/gordon-watchdog.sh
	76	+	```
	77	+
	78	+	Test it:
	79	+
	80	+	```bash
	81	+	~/.local/bin/gordon-watchdog.sh
	82	+	# Expected: [2024-01-15T10:30:00+01:00] OK: all routes healthy
	83	+	```
	84	+
	85	+	## What Gets Flagged
	86	+
	87	+	\| Condition \| Example \|
	88	+	\|-----------\|---------\|
	89	+	\| HTTP 4xx or 5xx \| App returning 500 \|
	90	+	\| HTTP 0 \| Container not responding at all \|
	91	+	\| Container not running \| Crashed or stopped container \|
	92	+	\| Attachment not running \| Database sidecar down \|
	93	+
	94	+	Redirects (301, 302, 303, 307) are not flagged. These are normal for apps that redirect HTTP to HTTPS or to a login page.
	95	+
	96	+	## systemd Timer
	97	+
	98	+	Two unit files go in `~/.config/systemd/user/`.
	99	+
	100	+	### Service unit
	101	+
	102	+	`gordon-watchdog.service`:
	103	+
	104	+	```ini
	105	+	[Unit]
	106	+	Description=Gordon Routes Health Check
	107	+	After=network-online.target
	108	+
	109	+	[Service]
	110	+	Type=oneshot
	111	+	ExecStart=%h/.local/bin/gordon-watchdog.sh
	112	+	Environment=DISPLAY=:0
	113	+	Environment=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/%U/bus
	114	+	```
	115	+
	116	+	### Timer unit
	117	+
	118	+	`gordon-watchdog.timer`:
	119	+
	120	+	```ini
	121	+	[Unit]
	122	+	Description=Run Gordon Watchdog every 5 minutes
	123	+
	124	+	[Timer]
	125	+	OnBootSec=1min
	126	+	OnUnitActiveSec=5min
	127	+	Persistent=true
	128	+
	129	+	[Install]
	130	+	WantedBy=timers.target
	131	+	```
	132	+
	133	+	### Enable it
	134	+
	135	+	```bash
	136	+	systemctl --user daemon-reload
	137	+	systemctl --user enable --now gordon-watchdog.timer
	138	+	```
	139	+
	140	+	### Verify
	141	+
	142	+	```bash
	143	+	# Timer is active
	144	+	systemctl --user status gordon-watchdog.timer
	145	+
	146	+	# Run it manually
	147	+	systemctl --user start gordon-watchdog.service
	148	+
	149	+	# Check logs
	150	+	journalctl --user -u gordon-watchdog.service --since "1 hour ago"
	151	+	```
	152	+
	153	+	---
	154	+
	155	+	## Customization
	156	+
	157	+	Change the interval: edit `OnUnitActiveSec` in the timer file. Use `1min`, `10min`, `30min`, etc.
	158	+
	159	+	Log to a file: add output redirection in the service unit:
	160	+
	161	+	```ini
	162	+	StandardOutput=append:%h/.local/share/gordon-watchdog.log
	163	+	StandardError=append:%h/.local/share/gordon-watchdog.log
	164	+	```
	165	+
	166	+	Send alerts elsewhere: replace `notify-send` in the script with whatever you need. `curl` a webhook, write to a Slack channel, send an email. The `$body` variable has the list of problems, one per line.
	167	+
	168	+	Stricter HTTP checks: if you want to flag anything that isn't exactly 200, replace the HTTP check with:
	169	+
	170	+	```bash
	171	+	if [[ "$http_status" -ne 200 ]]; then
	172	+	problems+=("$domain: HTTP $http_status")
	173	+	fi
	174	+	```
	175	+
	176	+	Remote monitoring: set `GORDON_CMD` to include `--remote` and `--token` flags, or export `GORDON_TOKEN` as an environment variable in the service unit.
	177	+
	178	+	---
	179	+
	180	+	## Troubleshooting
	181	+
	182	+	### No desktop notification
	183	+
	184	+	The systemd user service needs access to the D-Bus session bus and display. If notifications don't appear:
	185	+
	186	+	```bash
	187	+	# Check your UID
	188	+	id -u
	189	+	# Should match the %U in the service file (usually 1000)
	190	+
	191	+	# Verify D-Bus socket exists
	192	+	ls /run/user/$(id -u)/bus
	193	+	```
	194	+
	195	+	On Wayland, you may also need `WAYLAND_DISPLAY` in the service environment.
	196	+
	197	+	### "Cannot reach Gordon" on every run
	198	+
	199	+	If Gordon runs as a user service, ensure it's started before the watchdog. Add to the service unit:
	200	+
	201	+	```ini
	202	+	[Unit]
	203	+	After=network-online.target gordon.service
	204	+	```
	205	+
	206	+	### Timer not firing
	207	+
	208	+	```bash
	209	+	systemctl --user list-timers --all \| grep gordon
	210	+	```
	211	+
	212	+	If the timer shows `n/a` for next run, re-enable it:
	213	+
	214	+	```bash
	215	+	systemctl --user reenable gordon-watchdog.timer
	216	+	systemctl --user start gordon-watchdog.timer
	217	+	```

M wiki/guides/index.md +4 -0


@@ -17,6 +17,10 @@ In-depth guides for specific Gordon setups and integrations.
17	17
18	18		- [Remote CLI Management](./remote-cli.md) - Manage Gordon instances remotely
19	19
	20	+	## Monitoring
	21	+
	22	+	- [Health Monitoring with systemd](./health-monitoring.md) - Desktop alerts when routes go down
	23	+
20	24		## Secrets Management
21	25
22	26		- [Using Pass for Secrets](./secrets-pass.md) - Unix password manager integration


@@ -17,6 +17,10 @@ In-depth guides for specific Gordon setups and integrations.
17		17
18	- [Remote CLI Management](./remote-cli.md) - Manage Gordon instances remotely	18		- [Remote CLI Management](./remote-cli.md) - Manage Gordon instances remotely
19		19
		20	+	## Monitoring
		21	+
		22	+	- [Health Monitoring with systemd](./health-monitoring.md) - Desktop alerts when routes go down
		23	+
20	## Secrets Management	24		## Secrets Management
21		25
22	- [Using Pass for Secrets](./secrets-pass.md) - Unix password manager integration	26		- [Using Pass for Secrets](./secrets-pass.md) - Unix password manager integration

M wiki/index.md +1 -0


@@ -21,6 +21,7 @@ In-depth guides for specific setups:
21	21		- [Remote CLI Management](./guides/remote-cli.md) - Manage Gordon instances remotely
22	22		- [Using Pass for Secrets](./guides/secrets-pass.md) - Unix password manager integration
23	23		- [Using SOPS for Secrets](./guides/secrets-sops.md) - Encrypted secrets with SOPS
	24	+	- [Health Monitoring with systemd](./guides/health-monitoring.md) - Desktop alerts when routes go down
24	25
25	26		## Agents
26	27


@@ -21,6 +21,7 @@ In-depth guides for specific setups:
21	- [Remote CLI Management](./guides/remote-cli.md) - Manage Gordon instances remotely	21		- [Remote CLI Management](./guides/remote-cli.md) - Manage Gordon instances remotely
22	- [Using Pass for Secrets](./guides/secrets-pass.md) - Unix password manager integration	22		- [Using Pass for Secrets](./guides/secrets-pass.md) - Unix password manager integration
23	- [Using SOPS for Secrets](./guides/secrets-sops.md) - Encrypted secrets with SOPS	23		- [Using SOPS for Secrets](./guides/secrets-sops.md) - Encrypted secrets with SOPS
		24	+	- [Health Monitoring with systemd](./guides/health-monitoring.md) - Desktop alerts when routes go down
24		25
25	## Agents	26		## Agents
26		27