Merge pull request #145 from bnema/wiki/health-monitoring-guide

docs(wiki): health monitoring with systemd

commit: 372b09b1d6dff53383690c59db196b883725c0b4
author: bnema · 7mo ago · 2026-03-24 05:38 UTC
committer: GitHub · 7mo ago · 2026-03-24 05:38 UTC
merge of: 15d9fe3 + 618e538 · diff against the first parent
signature: GPG UnverifiedB5690EEEBB952194 The key is unknown

3 files · +228 -0

A wiki/guides/health-monitoring.md +223 -0

@@ -0,0 +1,223 @@
	1	+	# Health Monitoring with systemd
	2	+
	3	+	Monitor your Gordon routes and get desktop notifications when something goes down. Uses `gordon routes list --json`, a bash script, and a systemd user timer.
	4	+
	5	+	## Prerequisites
	6	+
	7	+	- Gordon CLI installed
	8	+	- `jq` and `notify-send` (libnotify) available
	9	+	- systemd user session (most Linux desktops)
	10	+
	11	+	For remote instances, configure a saved remote with `gordon remotes add` and `gordon remotes use` (see [Remote CLI Management](./remote-cli.md)). The CLI uses the active remote automatically.
	12	+
	13	+	---
	14	+
	15	+	## The Check Script
	16	+
	17	+	Create `~/.local/bin/gordon-watchdog.sh`:
	18	+
	19	+	```bash
	20	+	#!/usr/bin/env bash
	21	+	set -euo pipefail
	22	+
	23	+	GORDON_CMD="gordon routes list --json"
	24	+	# For remote monitoring, configure a saved remote first:
	25	+	# gordon remotes add myserver https://gordon.example.com --token-env GORDON_TOKEN
	26	+	# gordon remotes use myserver
	27	+	# The CLI then targets the remote automatically.
	28	+
	29	+	json=$($GORDON_CMD) \|\| {
	30	+	notify-send -u critical "Gordon Watchdog" "Cannot reach Gordon"
	31	+	exit 1
	32	+	}
	33	+
	34	+	problems=()
	35	+
	36	+	while IFS= read -r route; do
	37	+	domain=$(echo "$route" \| jq -r '.domain')
	38	+	http_status=$(echo "$route" \| jq -r '.http_status')
	39	+	container_status=$(echo "$route" \| jq -r '.container_status')
	40	+
	41	+	# HTTP status: flag 4xx, 5xx, 0 (no response), or unavailable
	42	+	if ! [[ "$http_status" =~ ^[0-9]+$ ]]; then
	43	+	problems+=("$domain: HTTP status unavailable ($http_status)")
	44	+	elif [[ "$http_status" -eq 0 ]] \|\| [[ "$http_status" -ge 400 ]]; then
	45	+	problems+=("$domain: HTTP $http_status")
	46	+	fi
	47	+
	48	+	# Container not running
	49	+	if [[ "$container_status" != "running" ]]; then
	50	+	problems+=("$domain: container $container_status")
	51	+	fi
	52	+
	53	+	# Check attachments (databases, sidecars)
	54	+	while IFS= read -r att; do
	55	+	att_name=$(echo "$att" \| jq -r '.name')
	56	+	att_status=$(echo "$att" \| jq -r '.status')
	57	+	if [[ "$att_status" != "running" ]]; then
	58	+	problems+=("$domain/$att_name: $att_status")
	59	+	fi
	60	+	done < <(echo "$route" \| jq -c '.attachments[]? // empty')
	61	+
	62	+	done < <(echo "$json" \| jq -c '.[]')
	63	+
	64	+	if [[ ${#problems[@]} -gt 0 ]]; then
	65	+	body=$(printf '%s\n' "${problems[@]}")
	66	+	notify-send -u critical -i dialog-warning \
	67	+	"Gordon Watchdog" \
	68	+	"${#problems[@]} problem(s):\n$body"
	69	+	echo "[$(date -Iseconds)] ALERT: ${#problems[@]} problem(s)"
	70	+	printf ' %s\n' "${problems[@]}"
	71	+	else
	72	+	echo "[$(date -Iseconds)] OK: all routes healthy"
	73	+	fi
	74	+	```
	75	+
	76	+	Make it executable:
	77	+
	78	+	```bash
	79	+	chmod +x ~/.local/bin/gordon-watchdog.sh
	80	+	```
	81	+
	82	+	Test it:
	83	+
	84	+	```bash
	85	+	~/.local/bin/gordon-watchdog.sh
	86	+	# Expected: [YYYY-MM-DDThh:mm:ss+00:00] OK: all routes healthy
	87	+	```
	88	+
	89	+	## What Gets Flagged
	90	+
	91	+	\| Condition \| Example \|
	92	+	\|-----------\|---------\|
	93	+	\| HTTP 4xx or 5xx \| App returning 500 \|
	94	+	\| HTTP 0 \| Container not responding at all \|
	95	+	\| Container not running \| Crashed or stopped container \|
	96	+	\| Attachment not running \| Database sidecar down \|
	97	+
	98	+	Redirects (301, 302, 303, 307) are not flagged. These are normal for apps that redirect HTTP to HTTPS or to a login page.
	99	+
	100	+	## systemd Timer
	101	+
	102	+	Two unit files go in `~/.config/systemd/user/`.
	103	+
	104	+	### Service unit
	105	+
	106	+	`gordon-watchdog.service`:
	107	+
	108	+	```ini
	109	+	[Unit]
	110	+	Description=Gordon Routes Health Check
	111	+	After=network-online.target
	112	+
	113	+	[Service]
	114	+	Type=oneshot
	115	+	ExecStart=%h/.local/bin/gordon-watchdog.sh
	116	+	Environment=DISPLAY=:0
	117	+	Environment=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/%U/bus
	118	+	```
	119	+
	120	+	### Timer unit
	121	+
	122	+	`gordon-watchdog.timer`:
	123	+
	124	+	```ini
	125	+	[Unit]
	126	+	Description=Run Gordon Watchdog every 5 minutes
	127	+
	128	+	[Timer]
	129	+	OnBootSec=1min
	130	+	OnUnitActiveSec=5min
	131	+	Persistent=true
	132	+
	133	+	[Install]
	134	+	WantedBy=timers.target
	135	+	```
	136	+
	137	+	### Enable it
	138	+
	139	+	```bash
	140	+	systemctl --user daemon-reload
	141	+	systemctl --user enable --now gordon-watchdog.timer
	142	+	```
	143	+
	144	+	### Verify
	145	+
	146	+	```bash
	147	+	# Timer is active
	148	+	systemctl --user status gordon-watchdog.timer
	149	+
	150	+	# Run it manually
	151	+	systemctl --user start gordon-watchdog.service
	152	+
	153	+	# Check logs
	154	+	journalctl --user -u gordon-watchdog.service --since "1 hour ago"
	155	+	```
	156	+
	157	+	---
	158	+
	159	+	## Customization
	160	+
	161	+	Change the interval: edit `OnUnitActiveSec` in the timer file. Use `1min`, `10min`, `30min`, etc.
	162	+
	163	+	Log to a file: add output redirection in the service unit:
	164	+
	165	+	```ini
	166	+	StandardOutput=append:%h/.local/share/gordon-watchdog.log
	167	+	StandardError=append:%h/.local/share/gordon-watchdog.log
	168	+	```
	169	+
	170	+	Send alerts elsewhere: replace `notify-send` in the script with whatever you need. `curl` a webhook, write to a Slack channel, send an email. The `$body` variable has the list of problems, one per line.
	171	+
	172	+	Stricter HTTP checks: if you want to flag anything that isn't exactly 200, replace the HTTP check with:
	173	+
	174	+	```bash
	175	+	if ! [[ "$http_status" =~ ^[0-9]+$ ]]; then
	176	+	problems+=("$domain: HTTP status unavailable ($http_status)")
	177	+	elif [[ "$http_status" -ne 200 ]]; then
	178	+	problems+=("$domain: HTTP $http_status")
	179	+	fi
	180	+	```
	181	+
	182	+	Remote monitoring: configure a saved remote with `gordon remotes add` and `gordon remotes use`. The script picks it up automatically from `~/.config/gordon/remotes.toml`.
	183	+
	184	+	---
	185	+
	186	+	## Troubleshooting
	187	+
	188	+	### No desktop notification
	189	+
	190	+	The systemd user service needs access to the D-Bus session bus and display. If notifications don't appear:
	191	+
	192	+	```bash
	193	+	# Check your UID
	194	+	id -u
	195	+	# Should match the %U in the service file (usually 1000)
	196	+
	197	+	# Verify D-Bus socket exists
	198	+	ls /run/user/$(id -u)/bus
	199	+	```
	200	+
	201	+	On Wayland, you may also need `WAYLAND_DISPLAY` in the service environment.
	202	+
	203	+	### "Cannot reach Gordon" on every run
	204	+
	205	+	If Gordon runs as a user service, ensure it's started before the watchdog. Add to the service unit:
	206	+
	207	+	```ini
	208	+	[Unit]
	209	+	After=network-online.target gordon.service
	210	+	```
	211	+
	212	+	### Timer not firing
	213	+
	214	+	```bash
	215	+	systemctl --user list-timers --all \| grep gordon
	216	+	```
	217	+
	218	+	If the timer shows `n/a` for next run, re-enable it:
	219	+
	220	+	```bash
	221	+	systemctl --user reenable gordon-watchdog.timer
	222	+	systemctl --user start gordon-watchdog.timer
	223	+	```

@@ -0,0 +1,223 @@
	1	+	# Health Monitoring with systemd
	2	+
	3	+	Monitor your Gordon routes and get desktop notifications when something goes down. Uses `gordon routes list --json`, a bash script, and a systemd user timer.
	4	+
	5	+	## Prerequisites
	6	+
	7	+	- Gordon CLI installed
	8	+	- `jq` and `notify-send` (libnotify) available
	9	+	- systemd user session (most Linux desktops)
	10	+
	11	+	For remote instances, configure a saved remote with `gordon remotes add` and `gordon remotes use` (see [Remote CLI Management](./remote-cli.md)). The CLI uses the active remote automatically.
	12	+
	13	+	---
	14	+
	15	+	## The Check Script
	16	+
	17	+	Create `~/.local/bin/gordon-watchdog.sh`:
	18	+
	19	+	```bash
	20	+	#!/usr/bin/env bash
	21	+	set -euo pipefail
	22	+
	23	+	GORDON_CMD="gordon routes list --json"
	24	+	# For remote monitoring, configure a saved remote first:
	25	+	# gordon remotes add myserver https://gordon.example.com --token-env GORDON_TOKEN
	26	+	# gordon remotes use myserver
	27	+	# The CLI then targets the remote automatically.
	28	+
	29	+	json=$($GORDON_CMD) \|\| {
	30	+	notify-send -u critical "Gordon Watchdog" "Cannot reach Gordon"
	31	+	exit 1
	32	+	}
	33	+
	34	+	problems=()
	35	+
	36	+	while IFS= read -r route; do
	37	+	domain=$(echo "$route" \| jq -r '.domain')
	38	+	http_status=$(echo "$route" \| jq -r '.http_status')
	39	+	container_status=$(echo "$route" \| jq -r '.container_status')
	40	+
	41	+	# HTTP status: flag 4xx, 5xx, 0 (no response), or unavailable
	42	+	if ! [[ "$http_status" =~ ^[0-9]+$ ]]; then
	43	+	problems+=("$domain: HTTP status unavailable ($http_status)")
	44	+	elif [[ "$http_status" -eq 0 ]] \|\| [[ "$http_status" -ge 400 ]]; then
	45	+	problems+=("$domain: HTTP $http_status")
	46	+	fi
	47	+
	48	+	# Container not running
	49	+	if [[ "$container_status" != "running" ]]; then
	50	+	problems+=("$domain: container $container_status")
	51	+	fi
	52	+
	53	+	# Check attachments (databases, sidecars)
	54	+	while IFS= read -r att; do
	55	+	att_name=$(echo "$att" \| jq -r '.name')
	56	+	att_status=$(echo "$att" \| jq -r '.status')
	57	+	if [[ "$att_status" != "running" ]]; then
	58	+	problems+=("$domain/$att_name: $att_status")
	59	+	fi
	60	+	done < <(echo "$route" \| jq -c '.attachments[]? // empty')
	61	+
	62	+	done < <(echo "$json" \| jq -c '.[]')
	63	+
	64	+	if [[ ${#problems[@]} -gt 0 ]]; then
	65	+	body=$(printf '%s\n' "${problems[@]}")
	66	+	notify-send -u critical -i dialog-warning \
	67	+	"Gordon Watchdog" \
	68	+	"${#problems[@]} problem(s):\n$body"
	69	+	echo "[$(date -Iseconds)] ALERT: ${#problems[@]} problem(s)"
	70	+	printf ' %s\n' "${problems[@]}"
	71	+	else
	72	+	echo "[$(date -Iseconds)] OK: all routes healthy"
	73	+	fi
	74	+	```
	75	+
	76	+	Make it executable:
	77	+
	78	+	```bash
	79	+	chmod +x ~/.local/bin/gordon-watchdog.sh
	80	+	```
	81	+
	82	+	Test it:
	83	+
	84	+	```bash
	85	+	~/.local/bin/gordon-watchdog.sh
	86	+	# Expected: [YYYY-MM-DDThh:mm:ss+00:00] OK: all routes healthy
	87	+	```
	88	+
	89	+	## What Gets Flagged
	90	+
	91	+	\| Condition \| Example \|
	92	+	\|-----------\|---------\|
	93	+	\| HTTP 4xx or 5xx \| App returning 500 \|
	94	+	\| HTTP 0 \| Container not responding at all \|
	95	+	\| Container not running \| Crashed or stopped container \|
	96	+	\| Attachment not running \| Database sidecar down \|
	97	+
	98	+	Redirects (301, 302, 303, 307) are not flagged. These are normal for apps that redirect HTTP to HTTPS or to a login page.
	99	+
	100	+	## systemd Timer
	101	+
	102	+	Two unit files go in `~/.config/systemd/user/`.
	103	+
	104	+	### Service unit
	105	+
	106	+	`gordon-watchdog.service`:
	107	+
	108	+	```ini
	109	+	[Unit]
	110	+	Description=Gordon Routes Health Check
	111	+	After=network-online.target
	112	+
	113	+	[Service]
	114	+	Type=oneshot
	115	+	ExecStart=%h/.local/bin/gordon-watchdog.sh
	116	+	Environment=DISPLAY=:0
	117	+	Environment=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/%U/bus
	118	+	```
	119	+
	120	+	### Timer unit
	121	+
	122	+	`gordon-watchdog.timer`:
	123	+
	124	+	```ini
	125	+	[Unit]
	126	+	Description=Run Gordon Watchdog every 5 minutes
	127	+
	128	+	[Timer]
	129	+	OnBootSec=1min
	130	+	OnUnitActiveSec=5min
	131	+	Persistent=true
	132	+
	133	+	[Install]
	134	+	WantedBy=timers.target
	135	+	```
	136	+
	137	+	### Enable it
	138	+
	139	+	```bash
	140	+	systemctl --user daemon-reload
	141	+	systemctl --user enable --now gordon-watchdog.timer
	142	+	```
	143	+
	144	+	### Verify
	145	+
	146	+	```bash
	147	+	# Timer is active
	148	+	systemctl --user status gordon-watchdog.timer
	149	+
	150	+	# Run it manually
	151	+	systemctl --user start gordon-watchdog.service
	152	+
	153	+	# Check logs
	154	+	journalctl --user -u gordon-watchdog.service --since "1 hour ago"
	155	+	```
	156	+
	157	+	---
	158	+
	159	+	## Customization
	160	+
	161	+	Change the interval: edit `OnUnitActiveSec` in the timer file. Use `1min`, `10min`, `30min`, etc.
	162	+
	163	+	Log to a file: add output redirection in the service unit:
	164	+
	165	+	```ini
	166	+	StandardOutput=append:%h/.local/share/gordon-watchdog.log
	167	+	StandardError=append:%h/.local/share/gordon-watchdog.log
	168	+	```
	169	+
	170	+	Send alerts elsewhere: replace `notify-send` in the script with whatever you need. `curl` a webhook, write to a Slack channel, send an email. The `$body` variable has the list of problems, one per line.
	171	+
	172	+	Stricter HTTP checks: if you want to flag anything that isn't exactly 200, replace the HTTP check with:
	173	+
	174	+	```bash
	175	+	if ! [[ "$http_status" =~ ^[0-9]+$ ]]; then
	176	+	problems+=("$domain: HTTP status unavailable ($http_status)")
	177	+	elif [[ "$http_status" -ne 200 ]]; then
	178	+	problems+=("$domain: HTTP $http_status")
	179	+	fi
	180	+	```
	181	+
	182	+	Remote monitoring: configure a saved remote with `gordon remotes add` and `gordon remotes use`. The script picks it up automatically from `~/.config/gordon/remotes.toml`.
	183	+
	184	+	---
	185	+
	186	+	## Troubleshooting
	187	+
	188	+	### No desktop notification
	189	+
	190	+	The systemd user service needs access to the D-Bus session bus and display. If notifications don't appear:
	191	+
	192	+	```bash
	193	+	# Check your UID
	194	+	id -u
	195	+	# Should match the %U in the service file (usually 1000)
	196	+
	197	+	# Verify D-Bus socket exists
	198	+	ls /run/user/$(id -u)/bus
	199	+	```
	200	+
	201	+	On Wayland, you may also need `WAYLAND_DISPLAY` in the service environment.
	202	+
	203	+	### "Cannot reach Gordon" on every run
	204	+
	205	+	If Gordon runs as a user service, ensure it's started before the watchdog. Add to the service unit:
	206	+
	207	+	```ini
	208	+	[Unit]
	209	+	After=network-online.target gordon.service
	210	+	```
	211	+
	212	+	### Timer not firing
	213	+
	214	+	```bash
	215	+	systemctl --user list-timers --all \| grep gordon
	216	+	```
	217	+
	218	+	If the timer shows `n/a` for next run, re-enable it:
	219	+
	220	+	```bash
	221	+	systemctl --user reenable gordon-watchdog.timer
	222	+	systemctl --user start gordon-watchdog.timer
	223	+	```

M wiki/guides/index.md +4 -0


@@ -17,6 +17,10 @@ In-depth guides for specific Gordon setups and integrations.
17	17
18	18		- [Remote CLI Management](./remote-cli.md) - Manage Gordon instances remotely
19	19
	20	+	## Monitoring
	21	+
	22	+	- [Health Monitoring with systemd](./health-monitoring.md) - Desktop alerts when routes go down
	23	+
20	24		## Secrets Management
21	25
22	26		- [Using Pass for Secrets](./secrets-pass.md) - Unix password manager integration


@@ -17,6 +17,10 @@ In-depth guides for specific Gordon setups and integrations.
17		17
18	- [Remote CLI Management](./remote-cli.md) - Manage Gordon instances remotely	18		- [Remote CLI Management](./remote-cli.md) - Manage Gordon instances remotely
19		19
		20	+	## Monitoring
		21	+
		22	+	- [Health Monitoring with systemd](./health-monitoring.md) - Desktop alerts when routes go down
		23	+
20	## Secrets Management	24		## Secrets Management
21		25
22	- [Using Pass for Secrets](./secrets-pass.md) - Unix password manager integration	26		- [Using Pass for Secrets](./secrets-pass.md) - Unix password manager integration

M wiki/index.md +1 -0


@@ -21,6 +21,7 @@ In-depth guides for specific setups:
21	21		- [Remote CLI Management](./guides/remote-cli.md) - Manage Gordon instances remotely
22	22		- [Using Pass for Secrets](./guides/secrets-pass.md) - Unix password manager integration
23	23		- [Using SOPS for Secrets](./guides/secrets-sops.md) - Encrypted secrets with SOPS
	24	+	- [Health Monitoring with systemd](./guides/health-monitoring.md) - Desktop alerts when routes go down
24	25
25	26		## Agents
26	27


@@ -21,6 +21,7 @@ In-depth guides for specific setups:
21	- [Remote CLI Management](./guides/remote-cli.md) - Manage Gordon instances remotely	21		- [Remote CLI Management](./guides/remote-cli.md) - Manage Gordon instances remotely
22	- [Using Pass for Secrets](./guides/secrets-pass.md) - Unix password manager integration	22		- [Using Pass for Secrets](./guides/secrets-pass.md) - Unix password manager integration
23	- [Using SOPS for Secrets](./guides/secrets-sops.md) - Encrypted secrets with SOPS	23		- [Using SOPS for Secrets](./guides/secrets-sops.md) - Encrypted secrets with SOPS
		24	+	- [Health Monitoring with systemd](./guides/health-monitoring.md) - Desktop alerts when routes go down
24		25
25	## Agents	26		## Agents
26		27