docs(remote): document mobile UDP congestion resilience
- commit
- 7943a2afd76f82b74b5a6e8a9c032426f8014080
- author
- brice · · 2026-07-10 11:17 UTC
- parent
- 48a3b83
- signature
- GPG Verified
CD88F9E564E0102646FB6077A1A07C7C7C3E349D· arch-gpg Verified for the committer email
4 files · +128 -24
M docs/performance.md +4 -0
| @@ -24,6 +24,10 @@ strace -ff -c -o tmux-cat.strace tmux new-session -d -s perf-cat 'timeout 30s sh | |||
| 24 | 24 | ||
| 25 | 25 | Record host CPU, OS/kernel, terminal emulator, vev commit, tmux version, terminal size, workload duration, raw command lines, syscall summaries, bytes written per rendered frame, and heap profiles or allocation counts before comparing. | |
| 26 | 26 | ||
| 27 | + | ## UDP mobile congestion behavior | |
| 28 | + | ||
| 29 | + | The datagram transport uses one shared congestion-responsive byte pacer for initial sends and retransmits. Retransmissions therefore consume the same byte budget as new data instead of bypassing congestion feedback. Datagram clients use a one-state output window: the cumulative ACK releases that state, and newer repaint state coalesces while it remains unacknowledged rather than retaining an unbounded queue. | |
| 30 | + | ||
| 27 | 31 | Microbenchmarks executed locally for this review using `go test ./internal/adapters/ipc ./internal/usecase/daemon -run '^$' -bench=. -benchmem`: | |
| 28 | 32 | ||
| 29 | 33 | | benchmark | methodology | | |
| @@ -24,6 +24,10 @@ strace -ff -c -o tmux-cat.strace tmux new-session -d -s perf-cat 'timeout 30s sh | |||||
| 24 | 24 | ||||
| 25 | Record host CPU, OS/kernel, terminal emulator, vev commit, tmux version, terminal size, workload duration, raw command lines, syscall summaries, bytes written per rendered frame, and heap profiles or allocation counts before comparing. | 25 | Record host CPU, OS/kernel, terminal emulator, vev commit, tmux version, terminal size, workload duration, raw command lines, syscall summaries, bytes written per rendered frame, and heap profiles or allocation counts before comparing. | ||
| 26 | 26 | ||||
| 27 | + | ## UDP mobile congestion behavior | |||
| 28 | + | ||||
| 29 | + | The datagram transport uses one shared congestion-responsive byte pacer for initial sends and retransmits. Retransmissions therefore consume the same byte budget as new data instead of bypassing congestion feedback. Datagram clients use a one-state output window: the cumulative ACK releases that state, and newer repaint state coalesces while it remains unacknowledged rather than retaining an unbounded queue. | |||
| 30 | + | ||||
| 27 | Microbenchmarks executed locally for this review using `go test ./internal/adapters/ipc ./internal/usecase/daemon -run '^$' -bench=. -benchmem`: | 31 | Microbenchmarks executed locally for this review using `go test ./internal/adapters/ipc ./internal/usecase/daemon -run '^$' -bench=. -benchmem`: | ||
| 28 | 32 | ||||
| 29 | | benchmark | methodology | | 33 | | benchmark | methodology | | ||
M docs/remote-resilience.md +11 -5
| @@ -15,14 +15,20 @@ Resilience work uses these user-facing states: | |||
| 15 | 15 | ||
| 16 | 16 | | State | Meaning | | |
| 17 | 17 | | --- | --- | | |
| 18 | - | | `connected` | authenticated UDP traffic is flowing | | |
| 19 | - | | `degraded` | traffic is delayed or incomplete, but recovery is still expected | | |
| 20 | - | | `probing` | vev is trying alternate UDP paths before using SSH again | | |
| 18 | + | | `connected` | a complete record or meaningful ACK progress is recent | | |
| 19 | + | | `degraded` | authenticated UDP packets still arrive, but complete-record or ACK progress is delayed | | |
| 20 | + | | `probing` | authenticated UDP packet contact is absent, so vev probes for path recovery | | |
| 21 | 21 | | `resuming` | a replacement transport is attaching to the same session | | |
| 22 | - | | `offline` | no path currently works, but retries continue | | |
| 22 | + | | `offline` | authenticated UDP contact remains absent and vev resumes over SSH stdio | | |
| 23 | 23 | | `expired` | the session or recovery window is no longer available | | |
| 24 | 24 | ||
| 25 | - | When a terminal is in raw mode, recovery messages should appear as vev status text rather than as bytes written into the remote program's output stream. | |
| 25 | + | When a terminal is in raw mode, recovery messages appear as vev status text rather than as bytes written into the remote program's output stream. | |
| 26 | + | ||
| 27 | + | `VEV_REMOTE_TRANSPORT=stdio` is the SSH-stdio fallback: it keeps the session inside authenticated SSH when UDP is blocked or unavailable. It remains available while UDP recovery is attempted. | |
| 28 | + | ||
| 29 | + | ## UDP transport health diagnostics | |
| 30 | + | ||
| 31 | + | At debug logging, UDP transport-health lines report only state, elapsed progress ages, and bounded transport counters; they do not include payloads, addresses, keys, or identities. For a real remote attempt, `scripts/debug-remote-attach.sh --udp-health user@host` stores redacted transport-health lines and Linux `ss -u -m` socket-memory counters, then prints the exact collected file paths. | |
| 26 | 32 | ||
| 27 | 33 | ## Manual checks | |
| 28 | 34 | ||
| @@ -15,14 +15,20 @@ Resilience work uses these user-facing states: | |||||
| 15 | 15 | ||||
| 16 | | State | Meaning | | 16 | | State | Meaning | | ||
| 17 | | --- | --- | | 17 | | --- | --- | | ||
| 18 | - | | `connected` | authenticated UDP traffic is flowing | | 18 | + | | `connected` | a complete record or meaningful ACK progress is recent | |
| 19 | - | | `degraded` | traffic is delayed or incomplete, but recovery is still expected | | 19 | + | | `degraded` | authenticated UDP packets still arrive, but complete-record or ACK progress is delayed | |
| 20 | - | | `probing` | vev is trying alternate UDP paths before using SSH again | | 20 | + | | `probing` | authenticated UDP packet contact is absent, so vev probes for path recovery | |
| 21 | | `resuming` | a replacement transport is attaching to the same session | | 21 | | `resuming` | a replacement transport is attaching to the same session | | ||
| 22 | - | | `offline` | no path currently works, but retries continue | | 22 | + | | `offline` | authenticated UDP contact remains absent and vev resumes over SSH stdio | |
| 23 | | `expired` | the session or recovery window is no longer available | | 23 | | `expired` | the session or recovery window is no longer available | | ||
| 24 | 24 | ||||
| 25 | - | When a terminal is in raw mode, recovery messages should appear as vev status text rather than as bytes written into the remote program's output stream. | 25 | + | When a terminal is in raw mode, recovery messages appear as vev status text rather than as bytes written into the remote program's output stream. |
| 26 | + | ||||
| 27 | + | `VEV_REMOTE_TRANSPORT=stdio` is the SSH-stdio fallback: it keeps the session inside authenticated SSH when UDP is blocked or unavailable. It remains available while UDP recovery is attempted. | |||
| 28 | + | ||||
| 29 | + | ## UDP transport health diagnostics | |||
| 30 | + | ||||
| 31 | + | At debug logging, UDP transport-health lines report only state, elapsed progress ages, and bounded transport counters; they do not include payloads, addresses, keys, or identities. For a real remote attempt, `scripts/debug-remote-attach.sh --udp-health user@host` stores redacted transport-health lines and Linux `ss -u -m` socket-memory counters, then prints the exact collected file paths. | |||
| 26 | 32 | ||||
| 27 | ## Manual checks | 33 | ## Manual checks | ||
| 28 | 34 | ||||
M scripts/debug-remote-attach.sh +83 -19
| @@ -3,7 +3,7 @@ set -euo pipefail | |||
| 3 | 3 | ||
| 4 | 4 | usage() { | |
| 5 | 5 | cat <<'USAGE' >&2 | |
| 6 | - | Usage: scripts/debug-remote-attach.sh [--mode stdio|udp|default] [--session NAME] [--duration 8s] [--vev-bin PATH] user@host | |
| 6 | + | Usage: scripts/debug-remote-attach.sh [--mode stdio|udp|default] [--session NAME] [--duration 8s] [--vev-bin PATH] [--udp-health] user@host | |
| 7 | 7 | ||
| 8 | 8 | Runs a real remote attach smoke/debug attempt under a PTY, captures local and | |
| 9 | 9 | remote vev versions, and stores local/remote log tails around the attempt. | |
| @@ -19,6 +19,8 @@ Notes: | |||
| 19 | 19 | - Exit 124 from timeout is treated as a likely successful attach if no early | |
| 20 | 20 | vev/ssh error is captured; inspect the bundle for confirmation. | |
| 21 | 21 | - The remote host must be reachable by SSH for log collection. | |
| 22 | + | - --udp-health collects redacted UDP transport-health log lines and Linux | |
| 23 | + | ss -u -m socket-memory counters when available. | |
| 22 | 24 | - The attach attempt unsets VEV so the smoke test can run from inside vev. | |
| 23 | 25 | USAGE | |
| 24 | 26 | } | |
| @@ -27,6 +29,7 @@ mode="default" | |||
| 27 | 29 | session="" | |
| 28 | 30 | duration="8s" | |
| 29 | 31 | vev_bin="${VEV_BIN:-vev}" | |
| 32 | + | collect_udp_health=false | |
| 30 | 33 | while [[ $# -gt 0 ]]; do | |
| 31 | 34 | case "$1" in | |
| 32 | 35 | --mode) | |
| @@ -37,6 +40,8 @@ while [[ $# -gt 0 ]]; do | |||
| 37 | 40 | duration="${2:-}"; shift 2 ;; | |
| 38 | 41 | --vev-bin) | |
| 39 | 42 | vev_bin="${2:-}"; shift 2 ;; | |
| 43 | + | --udp-health) | |
| 44 | + | collect_udp_health=true; shift ;; | |
| 40 | 45 | -h|--help) | |
| 41 | 46 | usage; exit 0 ;; | |
| 42 | 47 | --) | |
| @@ -64,59 +69,110 @@ if [[ -n "$session" ]]; then | |||
| 64 | 69 | fi | |
| 65 | 70 | ||
| 66 | 71 | repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" | |
| 67 | - | out_dir="${TMPDIR:-/tmp}/vev-remote-attach-$target-$(date +%Y%m%d-%H%M%S)" | |
| 68 | - | out_dir="${out_dir//@/_}" | |
| 69 | - | out_dir="${out_dir//:/_}" | |
| 72 | + | out_dir="${TMPDIR:-/tmp}/vev-remote-attach-$(date +%Y%m%d-%H%M%S)" | |
| 70 | 73 | mkdir -p "$out_dir" | |
| 71 | 74 | ||
| 72 | 75 | local_state="${XDG_STATE_HOME:-$HOME/.local/state}/vev" | |
| 76 | + | diagnostic_files=() | |
| 77 | + | ||
| 78 | + | # All persisted and displayed command output passes through this filter. Keep | |
| 79 | + | # diagnostic counters while removing the target, local home, network addresses, | |
| 80 | + | # and common key material labels from bundles that may be shared. | |
| 81 | + | redact() { | |
| 82 | + | sed -E \ | |
| 83 | + | -e "s|${target//\/\\/\\}|[remote]|g" \ | |
| 84 | + | -e "s|${HOME//\/\\/\\}|[home]|g" \ | |
| 85 | + | -e 's/([[:alnum:]_.-]+@)?([[:alnum:]_.-]+\.)+[[:alpha:]]{2,}/[host]/g' \ | |
| 86 | + | -e 's/([0-9]{1,3}\.){3}[0-9]{1,3}/[address]/g' \ | |
| 87 | + | -e 's/([[:xdigit:]]{1,4}:){2,}[[:xdigit:]:]*/[address]/g' \ | |
| 88 | + | -e 's/(private[ _-]?key|ssh-rsa|ssh-ed25519|ecdsa-sha2-[^[:space:]]*)/[redacted-key]/Ig' | |
| 89 | + | } | |
| 90 | + | ||
| 91 | + | redact_file() { | |
| 92 | + | local raw="$1" dest="$2" | |
| 93 | + | redact <"$raw" >"$dest" || true | |
| 94 | + | rm -f "$raw" | |
| 95 | + | } | |
| 73 | 96 | ||
| 74 | 97 | run_capture() { | |
| 75 | - | local name="$1"; shift | |
| 98 | + | local name="$1" raw_out raw_err status | |
| 99 | + | shift | |
| 100 | + | raw_out="$out_dir/.$name.out.raw" | |
| 101 | + | raw_err="$out_dir/.$name.err.raw" | |
| 102 | + | status=0 | |
| 76 | 103 | { | |
| 77 | 104 | printf '$' | |
| 78 | 105 | printf ' %q' "$@" | |
| 79 | 106 | printf '\n' | |
| 80 | 107 | "$@" | |
| 81 | - | } >"$out_dir/$name.out" 2>"$out_dir/$name.err" || printf '%s\n' "$?" >"$out_dir/$name.exit" | |
| 108 | + | } >"$raw_out" 2>"$raw_err" || status=$? | |
| 109 | + | redact_file "$raw_out" "$out_dir/$name.out" | |
| 110 | + | redact_file "$raw_err" "$out_dir/$name.err" | |
| 111 | + | if [[ "$status" -ne 0 ]]; then printf '%s\n' "$status" >"$out_dir/$name.exit"; fi | |
| 82 | 112 | } | |
| 83 | 113 | ||
| 84 | 114 | copy_local_logs() { | |
| 85 | - | local label="$1" | |
| 115 | + | local label="$1" raw | |
| 86 | 116 | mkdir -p "$out_dir/local-$label" | |
| 87 | 117 | if [[ -d "$local_state" ]]; then | |
| 88 | 118 | find "$local_state" -maxdepth 1 -type f -name 'vev*.log' -print0 2>/dev/null | while IFS= read -r -d '' f; do | |
| 89 | - | tail -n 200 "$f" >"$out_dir/local-$label/$(basename "$f")" || true | |
| 119 | + | raw="$out_dir/local-$label/.$(basename "$f").raw" | |
| 120 | + | tail -n 200 "$f" >"$raw" || true | |
| 121 | + | redact_file "$raw" "$out_dir/local-$label/$(basename "$f")" | |
| 90 | 122 | done | |
| 91 | 123 | fi | |
| 92 | 124 | } | |
| 93 | 125 | ||
| 94 | 126 | copy_remote_logs() { | |
| 95 | - | local label="$1" | |
| 127 | + | local label="$1" raw_out="$out_dir/remote-$1/.logs.out.raw" raw_err="$out_dir/remote-$1/.logs.err.raw" | |
| 96 | 128 | mkdir -p "$out_dir/remote-$label" | |
| 97 | 129 | ssh -o BatchMode=yes -o ConnectTimeout=8 "$target" 'state="${XDG_STATE_HOME:-$HOME/.local/state}/vev"; if [ -d "$state" ]; then for f in "$state"/vev*.log; do [ -e "$f" ] || continue; echo "===== $f ====="; tail -n 200 "$f"; done; fi' \ | |
| 98 | - | >"$out_dir/remote-$label/logs.out" 2>"$out_dir/remote-$label/logs.err" || true | |
| 130 | + | >"$raw_out" 2>"$raw_err" || true | |
| 131 | + | redact_file "$raw_out" "$out_dir/remote-$label/logs.out" | |
| 132 | + | redact_file "$raw_err" "$out_dir/remote-$label/logs.err" | |
| 133 | + | } | |
| 134 | + | ||
| 135 | + | collect_udp_diagnostics() { | |
| 136 | + | local label="$1" local_dir="$out_dir/local-$1" remote_dir="$out_dir/remote-$1" | |
| 137 | + | if ! "$collect_udp_health"; then return; fi | |
| 138 | + | ||
| 139 | + | grep -hE 'udp( transport)? health' "$local_dir"/*.log 2>/dev/null | redact >"$local_dir/udp-transport-health.out" || true | |
| 140 | + | (ss -u -m 2>/dev/null || true) | awk '/skmem:/' | redact >"$local_dir/udp-socket-memory.out" | |
| 141 | + | diagnostic_files+=("$local_dir/udp-transport-health.out" "$local_dir/udp-socket-memory.out") | |
| 142 | + | ||
| 143 | + | ssh -o BatchMode=yes -o ConnectTimeout=8 "$target" 'state="${XDG_STATE_HOME:-$HOME/.local/state}/vev"; [ -d "$state" ] && grep -hE "udp( transport)? health" "$state"/vev*.log 2>/dev/null || true' \ | |
| 144 | + | 2>&1 | redact >"$remote_dir/udp-transport-health.out" || true | |
| 145 | + | ssh -o BatchMode=yes -o ConnectTimeout=8 "$target" 'ss -u -m 2>/dev/null | awk "/skmem:/" || true' \ | |
| 146 | + | 2>&1 | redact >"$remote_dir/udp-socket-memory.out" || true | |
| 147 | + | diagnostic_files+=("$remote_dir/udp-transport-health.out" "$remote_dir/udp-socket-memory.out") | |
| 99 | 148 | } | |
| 100 | 149 | ||
| 101 | 150 | printf 'debug bundle: %s\n' "$out_dir" | |
| 102 | - | printf 'target: %s\nattach_target: %s\nmode: %s\nduration: %s\n' "$target" "$attach_target" "$mode" "$duration" >"$out_dir/summary.txt" | |
| 151 | + | printf 'mode: %s\nduration: %s\n' "$mode" "$duration" >"$out_dir/summary.txt" | |
| 103 | 152 | ||
| 153 | + | raw_git="$out_dir/.git.raw" | |
| 104 | 154 | ( | |
| 105 | 155 | cd "$repo_root" | |
| 106 | 156 | git rev-parse --show-toplevel | |
| 107 | 157 | git rev-parse --short HEAD | |
| 108 | 158 | git status --short | |
| 109 | - | ) >"$out_dir/git.txt" 2>&1 || true | |
| 159 | + | ) >"$raw_git" 2>&1 || true | |
| 160 | + | redact_file "$raw_git" "$out_dir/git.txt" | |
| 110 | 161 | ||
| 111 | 162 | run_capture local-version "$vev_bin" --version | |
| 163 | + | raw_remote_version_out="$out_dir/.remote-version.out.raw" | |
| 164 | + | raw_remote_version_err="$out_dir/.remote-version.err.raw" | |
| 112 | 165 | ssh -o BatchMode=yes -o ConnectTimeout=8 "$target" 'set -e; command -v vev; vev --version; pgrep -a vev || true; state="${XDG_STATE_HOME:-$HOME/.local/state}/vev"; echo "state=$state"' \ | |
| 113 | - | >"$out_dir/remote-version.out" 2>"$out_dir/remote-version.err" || echo "$?" >"$out_dir/remote-version.exit" | |
| 166 | + | >"$raw_remote_version_out" 2>"$raw_remote_version_err" || echo "$?" >"$out_dir/remote-version.exit" | |
| 167 | + | redact_file "$raw_remote_version_out" "$out_dir/remote-version.out" | |
| 168 | + | redact_file "$raw_remote_version_err" "$out_dir/remote-version.err" | |
| 114 | 169 | ||
| 115 | 170 | copy_local_logs before | |
| 116 | 171 | copy_remote_logs before | |
| 117 | 172 | ||
| 118 | - | attach_log="$out_dir/attach.typescript" | |
| 119 | - | attach_err="$out_dir/attach.err" | |
| 173 | + | attach_log_raw="$out_dir/.attach.typescript.raw" | |
| 174 | + | attach_err_raw="$out_dir/.attach.err.raw" | |
| 175 | + | attach_stdout_raw="$out_dir/.attach.stdout.raw" | |
| 120 | 176 | cmd=("$vev_bin" attach "$attach_target") | |
| 121 | 177 | if [[ "$mode" == "default" ]]; then | |
| 122 | 178 | env_cmd=(env -u VEV -u VEV_REMOTE_TRANSPORT) | |
| @@ -127,16 +183,20 @@ fi | |||
| 127 | 183 | set +e | |
| 128 | 184 | if command -v script >/dev/null 2>&1; then | |
| 129 | 185 | printf -v attach_cmd '%q ' "${cmd[@]}" | |
| 130 | - | "${env_cmd[@]}" timeout "$duration" script -q -e -c "$attach_cmd" "$attach_log" >"$out_dir/attach.stdout" 2>"$attach_err" | |
| 186 | + | "${env_cmd[@]}" timeout "$duration" script -q -e -c "$attach_cmd" "$attach_log_raw" >"$attach_stdout_raw" 2>"$attach_err_raw" | |
| 131 | 187 | else | |
| 132 | - | "${env_cmd[@]}" timeout "$duration" "${cmd[@]}" >"$out_dir/attach.stdout" 2>"$attach_err" | |
| 188 | + | "${env_cmd[@]}" timeout "$duration" "${cmd[@]}" >"$attach_stdout_raw" 2>"$attach_err_raw" | |
| 133 | 189 | fi | |
| 134 | 190 | attach_status=$? | |
| 135 | 191 | set -e | |
| 192 | + | redact_file "$attach_stdout_raw" "$out_dir/attach.stdout" | |
| 193 | + | redact_file "$attach_err_raw" "$out_dir/attach.err" | |
| 194 | + | if [[ -e "$attach_log_raw" ]]; then redact_file "$attach_log_raw" "$out_dir/attach.typescript"; fi | |
| 136 | 195 | printf '%s\n' "$attach_status" >"$out_dir/attach.exit" | |
| 137 | 196 | ||
| 138 | 197 | copy_local_logs after | |
| 139 | 198 | copy_remote_logs after | |
| 199 | + | collect_udp_diagnostics after | |
| 140 | 200 | ||
| 141 | 201 | { | |
| 142 | 202 | echo "debug bundle: $out_dir" | |
| @@ -145,9 +205,13 @@ copy_remote_logs after | |||
| 145 | 205 | sed 's/^/ /' "$out_dir/local-version.out" 2>/dev/null || true | |
| 146 | 206 | echo "remote version:" | |
| 147 | 207 | sed 's/^/ /' "$out_dir/remote-version.out" 2>/dev/null || true | |
| 148 | - | if [[ -s "$attach_err" ]]; then | |
| 208 | + | if [[ -s "$out_dir/attach.err" ]]; then | |
| 149 | 209 | echo "attach stderr:" | |
| 150 | - | sed 's/^/ /' "$attach_err" | tail -40 | |
| 210 | + | sed 's/^/ /' "$out_dir/attach.err" | tail -40 | |
| 211 | + | fi | |
| 212 | + | if ((${#diagnostic_files[@]})); then | |
| 213 | + | echo "collected diagnostic files:" | |
| 214 | + | printf ' %s\n' "${diagnostic_files[@]}" | |
| 151 | 215 | fi | |
| 152 | 216 | echo "recent local after logs:" | |
| 153 | 217 | find "$out_dir/local-after" -maxdepth 1 -type f -print0 2>/dev/null | while IFS= read -r -d '' f; do | |
| @@ -3,7 +3,7 @@ set -euo pipefail | |||||
| 3 | 3 | ||||
| 4 | usage() { | 4 | usage() { | ||
| 5 | cat <<'USAGE' >&2 | 5 | cat <<'USAGE' >&2 | ||
| 6 | - | Usage: scripts/debug-remote-attach.sh [--mode stdio|udp|default] [--session NAME] [--duration 8s] [--vev-bin PATH] user@host | 6 | + | Usage: scripts/debug-remote-attach.sh [--mode stdio|udp|default] [--session NAME] [--duration 8s] [--vev-bin PATH] [--udp-health] user@host |
| 7 | 7 | ||||
| 8 | Runs a real remote attach smoke/debug attempt under a PTY, captures local and | 8 | Runs a real remote attach smoke/debug attempt under a PTY, captures local and | ||
| 9 | remote vev versions, and stores local/remote log tails around the attempt. | 9 | remote vev versions, and stores local/remote log tails around the attempt. | ||
| @@ -19,6 +19,8 @@ Notes: | |||||
| 19 | - Exit 124 from timeout is treated as a likely successful attach if no early | 19 | - Exit 124 from timeout is treated as a likely successful attach if no early | ||
| 20 | vev/ssh error is captured; inspect the bundle for confirmation. | 20 | vev/ssh error is captured; inspect the bundle for confirmation. | ||
| 21 | - The remote host must be reachable by SSH for log collection. | 21 | - The remote host must be reachable by SSH for log collection. | ||
| 22 | + | - --udp-health collects redacted UDP transport-health log lines and Linux | |||
| 23 | + | ss -u -m socket-memory counters when available. | |||
| 22 | - The attach attempt unsets VEV so the smoke test can run from inside vev. | 24 | - The attach attempt unsets VEV so the smoke test can run from inside vev. | ||
| 23 | USAGE | 25 | USAGE | ||
| 24 | } | 26 | } | ||
| @@ -27,6 +29,7 @@ mode="default" | |||||
| 27 | session="" | 29 | session="" | ||
| 28 | duration="8s" | 30 | duration="8s" | ||
| 29 | vev_bin="${VEV_BIN:-vev}" | 31 | vev_bin="${VEV_BIN:-vev}" | ||
| 32 | + | collect_udp_health=false | |||
| 30 | while [[ $# -gt 0 ]]; do | 33 | while [[ $# -gt 0 ]]; do | ||
| 31 | case "$1" in | 34 | case "$1" in | ||
| 32 | --mode) | 35 | --mode) | ||
| @@ -37,6 +40,8 @@ while [[ $# -gt 0 ]]; do | |||||
| 37 | duration="${2:-}"; shift 2 ;; | 40 | duration="${2:-}"; shift 2 ;; | ||
| 38 | --vev-bin) | 41 | --vev-bin) | ||
| 39 | vev_bin="${2:-}"; shift 2 ;; | 42 | vev_bin="${2:-}"; shift 2 ;; | ||
| 43 | + | --udp-health) | |||
| 44 | + | collect_udp_health=true; shift ;; | |||
| 40 | -h|--help) | 45 | -h|--help) | ||
| 41 | usage; exit 0 ;; | 46 | usage; exit 0 ;; | ||
| 42 | --) | 47 | --) | ||
| @@ -64,59 +69,110 @@ if [[ -n "$session" ]]; then | |||||
| 64 | fi | 69 | fi | ||
| 65 | 70 | ||||
| 66 | repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" | 71 | repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" | ||
| 67 | - | out_dir="${TMPDIR:-/tmp}/vev-remote-attach-$target-$(date +%Y%m%d-%H%M%S)" | 72 | + | out_dir="${TMPDIR:-/tmp}/vev-remote-attach-$(date +%Y%m%d-%H%M%S)" |
| 68 | - | out_dir="${out_dir//@/_}" | |||
| 69 | - | out_dir="${out_dir//:/_}" | |||
| 70 | mkdir -p "$out_dir" | 73 | mkdir -p "$out_dir" | ||
| 71 | 74 | ||||
| 72 | local_state="${XDG_STATE_HOME:-$HOME/.local/state}/vev" | 75 | local_state="${XDG_STATE_HOME:-$HOME/.local/state}/vev" | ||
| 76 | + | diagnostic_files=() | |||
| 77 | + | ||||
| 78 | + | # All persisted and displayed command output passes through this filter. Keep | |||
| 79 | + | # diagnostic counters while removing the target, local home, network addresses, | |||
| 80 | + | # and common key material labels from bundles that may be shared. | |||
| 81 | + | redact() { | |||
| 82 | + | sed -E \ | |||
| 83 | + | -e "s|${target//\/\\/\\}|[remote]|g" \ | |||
| 84 | + | -e "s|${HOME//\/\\/\\}|[home]|g" \ | |||
| 85 | + | -e 's/([[:alnum:]_.-]+@)?([[:alnum:]_.-]+\.)+[[:alpha:]]{2,}/[host]/g' \ | |||
| 86 | + | -e 's/([0-9]{1,3}\.){3}[0-9]{1,3}/[address]/g' \ | |||
| 87 | + | -e 's/([[:xdigit:]]{1,4}:){2,}[[:xdigit:]:]*/[address]/g' \ | |||
| 88 | + | -e 's/(private[ _-]?key|ssh-rsa|ssh-ed25519|ecdsa-sha2-[^[:space:]]*)/[redacted-key]/Ig' | |||
| 89 | + | } | |||
| 90 | + | ||||
| 91 | + | redact_file() { | |||
| 92 | + | local raw="$1" dest="$2" | |||
| 93 | + | redact <"$raw" >"$dest" || true | |||
| 94 | + | rm -f "$raw" | |||
| 95 | + | } | |||
| 73 | 96 | ||||
| 74 | run_capture() { | 97 | run_capture() { | ||
| 75 | - | local name="$1"; shift | 98 | + | local name="$1" raw_out raw_err status |
| 99 | + | shift | |||
| 100 | + | raw_out="$out_dir/.$name.out.raw" | |||
| 101 | + | raw_err="$out_dir/.$name.err.raw" | |||
| 102 | + | status=0 | |||
| 76 | { | 103 | { | ||
| 77 | printf '$' | 104 | printf '$' | ||
| 78 | printf ' %q' "$@" | 105 | printf ' %q' "$@" | ||
| 79 | printf '\n' | 106 | printf '\n' | ||
| 80 | "$@" | 107 | "$@" | ||
| 81 | - | } >"$out_dir/$name.out" 2>"$out_dir/$name.err" || printf '%s\n' "$?" >"$out_dir/$name.exit" | 108 | + | } >"$raw_out" 2>"$raw_err" || status=$? |
| 109 | + | redact_file "$raw_out" "$out_dir/$name.out" | |||
| 110 | + | redact_file "$raw_err" "$out_dir/$name.err" | |||
| 111 | + | if [[ "$status" -ne 0 ]]; then printf '%s\n' "$status" >"$out_dir/$name.exit"; fi | |||
| 82 | } | 112 | } | ||
| 83 | 113 | ||||
| 84 | copy_local_logs() { | 114 | copy_local_logs() { | ||
| 85 | - | local label="$1" | 115 | + | local label="$1" raw |
| 86 | mkdir -p "$out_dir/local-$label" | 116 | mkdir -p "$out_dir/local-$label" | ||
| 87 | if [[ -d "$local_state" ]]; then | 117 | if [[ -d "$local_state" ]]; then | ||
| 88 | find "$local_state" -maxdepth 1 -type f -name 'vev*.log' -print0 2>/dev/null | while IFS= read -r -d '' f; do | 118 | find "$local_state" -maxdepth 1 -type f -name 'vev*.log' -print0 2>/dev/null | while IFS= read -r -d '' f; do | ||
| 89 | - | tail -n 200 "$f" >"$out_dir/local-$label/$(basename "$f")" || true | 119 | + | raw="$out_dir/local-$label/.$(basename "$f").raw" |
| 120 | + | tail -n 200 "$f" >"$raw" || true | |||
| 121 | + | redact_file "$raw" "$out_dir/local-$label/$(basename "$f")" | |||
| 90 | done | 122 | done | ||
| 91 | fi | 123 | fi | ||
| 92 | } | 124 | } | ||
| 93 | 125 | ||||
| 94 | copy_remote_logs() { | 126 | copy_remote_logs() { | ||
| 95 | - | local label="$1" | 127 | + | local label="$1" raw_out="$out_dir/remote-$1/.logs.out.raw" raw_err="$out_dir/remote-$1/.logs.err.raw" |
| 96 | mkdir -p "$out_dir/remote-$label" | 128 | mkdir -p "$out_dir/remote-$label" | ||
| 97 | ssh -o BatchMode=yes -o ConnectTimeout=8 "$target" 'state="${XDG_STATE_HOME:-$HOME/.local/state}/vev"; if [ -d "$state" ]; then for f in "$state"/vev*.log; do [ -e "$f" ] || continue; echo "===== $f ====="; tail -n 200 "$f"; done; fi' \ | 129 | ssh -o BatchMode=yes -o ConnectTimeout=8 "$target" 'state="${XDG_STATE_HOME:-$HOME/.local/state}/vev"; if [ -d "$state" ]; then for f in "$state"/vev*.log; do [ -e "$f" ] || continue; echo "===== $f ====="; tail -n 200 "$f"; done; fi' \ | ||
| 98 | - | >"$out_dir/remote-$label/logs.out" 2>"$out_dir/remote-$label/logs.err" || true | 130 | + | >"$raw_out" 2>"$raw_err" || true |
| 131 | + | redact_file "$raw_out" "$out_dir/remote-$label/logs.out" | |||
| 132 | + | redact_file "$raw_err" "$out_dir/remote-$label/logs.err" | |||
| 133 | + | } | |||
| 134 | + | ||||
| 135 | + | collect_udp_diagnostics() { | |||
| 136 | + | local label="$1" local_dir="$out_dir/local-$1" remote_dir="$out_dir/remote-$1" | |||
| 137 | + | if ! "$collect_udp_health"; then return; fi | |||
| 138 | + | ||||
| 139 | + | grep -hE 'udp( transport)? health' "$local_dir"/*.log 2>/dev/null | redact >"$local_dir/udp-transport-health.out" || true | |||
| 140 | + | (ss -u -m 2>/dev/null || true) | awk '/skmem:/' | redact >"$local_dir/udp-socket-memory.out" | |||
| 141 | + | diagnostic_files+=("$local_dir/udp-transport-health.out" "$local_dir/udp-socket-memory.out") | |||
| 142 | + | ||||
| 143 | + | ssh -o BatchMode=yes -o ConnectTimeout=8 "$target" 'state="${XDG_STATE_HOME:-$HOME/.local/state}/vev"; [ -d "$state" ] && grep -hE "udp( transport)? health" "$state"/vev*.log 2>/dev/null || true' \ | |||
| 144 | + | 2>&1 | redact >"$remote_dir/udp-transport-health.out" || true | |||
| 145 | + | ssh -o BatchMode=yes -o ConnectTimeout=8 "$target" 'ss -u -m 2>/dev/null | awk "/skmem:/" || true' \ | |||
| 146 | + | 2>&1 | redact >"$remote_dir/udp-socket-memory.out" || true | |||
| 147 | + | diagnostic_files+=("$remote_dir/udp-transport-health.out" "$remote_dir/udp-socket-memory.out") | |||
| 99 | } | 148 | } | ||
| 100 | 149 | ||||
| 101 | printf 'debug bundle: %s\n' "$out_dir" | 150 | printf 'debug bundle: %s\n' "$out_dir" | ||
| 102 | - | printf 'target: %s\nattach_target: %s\nmode: %s\nduration: %s\n' "$target" "$attach_target" "$mode" "$duration" >"$out_dir/summary.txt" | 151 | + | printf 'mode: %s\nduration: %s\n' "$mode" "$duration" >"$out_dir/summary.txt" |
| 103 | 152 | ||||
| 153 | + | raw_git="$out_dir/.git.raw" | |||
| 104 | ( | 154 | ( | ||
| 105 | cd "$repo_root" | 155 | cd "$repo_root" | ||
| 106 | git rev-parse --show-toplevel | 156 | git rev-parse --show-toplevel | ||
| 107 | git rev-parse --short HEAD | 157 | git rev-parse --short HEAD | ||
| 108 | git status --short | 158 | git status --short | ||
| 109 | - | ) >"$out_dir/git.txt" 2>&1 || true | 159 | + | ) >"$raw_git" 2>&1 || true |
| 160 | + | redact_file "$raw_git" "$out_dir/git.txt" | |||
| 110 | 161 | ||||
| 111 | run_capture local-version "$vev_bin" --version | 162 | run_capture local-version "$vev_bin" --version | ||
| 163 | + | raw_remote_version_out="$out_dir/.remote-version.out.raw" | |||
| 164 | + | raw_remote_version_err="$out_dir/.remote-version.err.raw" | |||
| 112 | ssh -o BatchMode=yes -o ConnectTimeout=8 "$target" 'set -e; command -v vev; vev --version; pgrep -a vev || true; state="${XDG_STATE_HOME:-$HOME/.local/state}/vev"; echo "state=$state"' \ | 165 | ssh -o BatchMode=yes -o ConnectTimeout=8 "$target" 'set -e; command -v vev; vev --version; pgrep -a vev || true; state="${XDG_STATE_HOME:-$HOME/.local/state}/vev"; echo "state=$state"' \ | ||
| 113 | - | >"$out_dir/remote-version.out" 2>"$out_dir/remote-version.err" || echo "$?" >"$out_dir/remote-version.exit" | 166 | + | >"$raw_remote_version_out" 2>"$raw_remote_version_err" || echo "$?" >"$out_dir/remote-version.exit" |
| 167 | + | redact_file "$raw_remote_version_out" "$out_dir/remote-version.out" | |||
| 168 | + | redact_file "$raw_remote_version_err" "$out_dir/remote-version.err" | |||
| 114 | 169 | ||||
| 115 | copy_local_logs before | 170 | copy_local_logs before | ||
| 116 | copy_remote_logs before | 171 | copy_remote_logs before | ||
| 117 | 172 | ||||
| 118 | - | attach_log="$out_dir/attach.typescript" | 173 | + | attach_log_raw="$out_dir/.attach.typescript.raw" |
| 119 | - | attach_err="$out_dir/attach.err" | 174 | + | attach_err_raw="$out_dir/.attach.err.raw" |
| 175 | + | attach_stdout_raw="$out_dir/.attach.stdout.raw" | |||
| 120 | cmd=("$vev_bin" attach "$attach_target") | 176 | cmd=("$vev_bin" attach "$attach_target") | ||
| 121 | if [[ "$mode" == "default" ]]; then | 177 | if [[ "$mode" == "default" ]]; then | ||
| 122 | env_cmd=(env -u VEV -u VEV_REMOTE_TRANSPORT) | 178 | env_cmd=(env -u VEV -u VEV_REMOTE_TRANSPORT) | ||
| @@ -127,16 +183,20 @@ fi | |||||
| 127 | set +e | 183 | set +e | ||
| 128 | if command -v script >/dev/null 2>&1; then | 184 | if command -v script >/dev/null 2>&1; then | ||
| 129 | printf -v attach_cmd '%q ' "${cmd[@]}" | 185 | printf -v attach_cmd '%q ' "${cmd[@]}" | ||
| 130 | - | "${env_cmd[@]}" timeout "$duration" script -q -e -c "$attach_cmd" "$attach_log" >"$out_dir/attach.stdout" 2>"$attach_err" | 186 | + | "${env_cmd[@]}" timeout "$duration" script -q -e -c "$attach_cmd" "$attach_log_raw" >"$attach_stdout_raw" 2>"$attach_err_raw" |
| 131 | else | 187 | else | ||
| 132 | - | "${env_cmd[@]}" timeout "$duration" "${cmd[@]}" >"$out_dir/attach.stdout" 2>"$attach_err" | 188 | + | "${env_cmd[@]}" timeout "$duration" "${cmd[@]}" >"$attach_stdout_raw" 2>"$attach_err_raw" |
| 133 | fi | 189 | fi | ||
| 134 | attach_status=$? | 190 | attach_status=$? | ||
| 135 | set -e | 191 | set -e | ||
| 192 | + | redact_file "$attach_stdout_raw" "$out_dir/attach.stdout" | |||
| 193 | + | redact_file "$attach_err_raw" "$out_dir/attach.err" | |||
| 194 | + | if [[ -e "$attach_log_raw" ]]; then redact_file "$attach_log_raw" "$out_dir/attach.typescript"; fi | |||
| 136 | printf '%s\n' "$attach_status" >"$out_dir/attach.exit" | 195 | printf '%s\n' "$attach_status" >"$out_dir/attach.exit" | ||
| 137 | 196 | ||||
| 138 | copy_local_logs after | 197 | copy_local_logs after | ||
| 139 | copy_remote_logs after | 198 | copy_remote_logs after | ||
| 199 | + | collect_udp_diagnostics after | |||
| 140 | 200 | ||||
| 141 | { | 201 | { | ||
| 142 | echo "debug bundle: $out_dir" | 202 | echo "debug bundle: $out_dir" | ||
| @@ -145,9 +205,13 @@ copy_remote_logs after | |||||
| 145 | sed 's/^/ /' "$out_dir/local-version.out" 2>/dev/null || true | 205 | sed 's/^/ /' "$out_dir/local-version.out" 2>/dev/null || true | ||
| 146 | echo "remote version:" | 206 | echo "remote version:" | ||
| 147 | sed 's/^/ /' "$out_dir/remote-version.out" 2>/dev/null || true | 207 | sed 's/^/ /' "$out_dir/remote-version.out" 2>/dev/null || true | ||
| 148 | - | if [[ -s "$attach_err" ]]; then | 208 | + | if [[ -s "$out_dir/attach.err" ]]; then |
| 149 | echo "attach stderr:" | 209 | echo "attach stderr:" | ||
| 150 | - | sed 's/^/ /' "$attach_err" | tail -40 | 210 | + | sed 's/^/ /' "$out_dir/attach.err" | tail -40 |
| 211 | + | fi | |||
| 212 | + | if ((${#diagnostic_files[@]})); then | |||
| 213 | + | echo "collected diagnostic files:" | |||
| 214 | + | printf ' %s\n' "${diagnostic_files[@]}" | |||
| 151 | fi | 215 | fi | ||
| 152 | echo "recent local after logs:" | 216 | echo "recent local after logs:" | ||
| 153 | find "$out_dir/local-after" -maxdepth 1 -type f -print0 2>/dev/null | while IFS= read -r -d '' f; do | 217 | find "$out_dir/local-after" -maxdepth 1 -type f -print0 2>/dev/null | while IFS= read -r -d '' f; do | ||
A scripts/debug-remote-attach_test.sh +30 -0
| @@ -0,0 +1,30 @@ | |||
| 1 | + | #!/usr/bin/env bash | |
| 2 | + | set -euo pipefail | |
| 3 | + | ||
| 4 | + | repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" | |
| 5 | + | help="$($repo_root/scripts/debug-remote-attach.sh --help 2>&1)" | |
| 6 | + | grep -Fq -- '--udp-health' <<<"$help" | |
| 7 | + | ||
| 8 | + | work="$(mktemp -d)" | |
| 9 | + | trap 'rm -rf "$work"' EXIT | |
| 10 | + | mkdir -p "$work/bin" "$work/state/vev" | |
| 11 | + | printf 'udp health peer=192.0.2.1 home=%s\n' "$HOME" >"$work/state/vev/vev-test.log" | |
| 12 | + | cat >"$work/bin/ssh" <<'SSH' | |
| 13 | + | #!/usr/bin/env bash | |
| 14 | + | printf 'udp transport health peer=198.51.100.2 key=ssh-ed25519\nskmem:(r0,rb1,t0,tb1,f0,w0,o0,bl0,d0)\n' | |
| 15 | + | SSH | |
| 16 | + | chmod +x "$work/bin/ssh" | |
| 17 | + | ||
| 18 | + | TMPDIR="$work" XDG_STATE_HOME="$work/state" PATH="$work/bin:$PATH" \ | |
| 19 | + | "$repo_root/scripts/debug-remote-attach.sh" --udp-health --duration 0s --vev-bin /bin/true alice@mobile.example.com >"$work/run.out" 2>&1 | |
| 20 | + | bundle="$(find "$work" -maxdepth 1 -type d -name 'vev-remote-attach-*' -print -quit)" | |
| 21 | + | test -n "$bundle" | |
| 22 | + | test -f "$bundle/local-after/udp-transport-health.out" | |
| 23 | + | test -f "$bundle/remote-after/udp-socket-memory.out" | |
| 24 | + | grep -Fq "$bundle/remote-after/udp-socket-memory.out" "$work/run.out" | |
| 25 | + | if grep -R --exclude='*.raw' -E 'alice@mobile\.example\.com|192\.0\.2\.1|198\.51\.100\.2|ssh-ed25519' "$bundle"; then | |
| 26 | + | echo 'unredacted diagnostic value found' >&2 | |
| 27 | + | exit 1 | |
| 28 | + | fi | |
| 29 | + | ||
| 30 | + | echo 'debug-remote-attach diagnostics/redaction: ok' | |
| @@ -0,0 +1,30 @@ | |||||
| 1 | + | #!/usr/bin/env bash | |||
| 2 | + | set -euo pipefail | |||
| 3 | + | ||||
| 4 | + | repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" | |||
| 5 | + | help="$($repo_root/scripts/debug-remote-attach.sh --help 2>&1)" | |||
| 6 | + | grep -Fq -- '--udp-health' <<<"$help" | |||
| 7 | + | ||||
| 8 | + | work="$(mktemp -d)" | |||
| 9 | + | trap 'rm -rf "$work"' EXIT | |||
| 10 | + | mkdir -p "$work/bin" "$work/state/vev" | |||
| 11 | + | printf 'udp health peer=192.0.2.1 home=%s\n' "$HOME" >"$work/state/vev/vev-test.log" | |||
| 12 | + | cat >"$work/bin/ssh" <<'SSH' | |||
| 13 | + | #!/usr/bin/env bash | |||
| 14 | + | printf 'udp transport health peer=198.51.100.2 key=ssh-ed25519\nskmem:(r0,rb1,t0,tb1,f0,w0,o0,bl0,d0)\n' | |||
| 15 | + | SSH | |||
| 16 | + | chmod +x "$work/bin/ssh" | |||
| 17 | + | ||||
| 18 | + | TMPDIR="$work" XDG_STATE_HOME="$work/state" PATH="$work/bin:$PATH" \ | |||
| 19 | + | "$repo_root/scripts/debug-remote-attach.sh" --udp-health --duration 0s --vev-bin /bin/true alice@mobile.example.com >"$work/run.out" 2>&1 | |||
| 20 | + | bundle="$(find "$work" -maxdepth 1 -type d -name 'vev-remote-attach-*' -print -quit)" | |||
| 21 | + | test -n "$bundle" | |||
| 22 | + | test -f "$bundle/local-after/udp-transport-health.out" | |||
| 23 | + | test -f "$bundle/remote-after/udp-socket-memory.out" | |||
| 24 | + | grep -Fq "$bundle/remote-after/udp-socket-memory.out" "$work/run.out" | |||
| 25 | + | if grep -R --exclude='*.raw' -E 'alice@mobile\.example\.com|192\.0\.2\.1|198\.51\.100\.2|ssh-ed25519' "$bundle"; then | |||
| 26 | + | echo 'unredacted diagnostic value found' >&2 | |||
| 27 | + | exit 1 | |||
| 28 | + | fi | |||
| 29 | + | ||||
| 30 | + | echo 'debug-remote-attach diagnostics/redaction: ok' | |||