secure-vps-setup.md Raw History Folder

Secure VPS Setup

A production-ready VPS security configuration with Tailscale-only SSH access and CrowdSec intrusion prevention.

Overview

This guide configures three security layers:

Layer	Protection
SSH	Tailscale only (invisible to internet)
Firewall	firewalld with DROP policy
IPS	CrowdSec with community blocklists

Prerequisites

Fresh Ubuntu 24.04 VPS
Tailscale account
Initial SSH access via public IP (temporary)
Cloudflare account for HTTPS termination

Step 1: Initial System Hardening

Before anything else, enable automatic security updates and time sync:

# Automatic security updates
apt update && apt install -y unattended-upgrades
dpkg-reconfigure -plow unattended-upgrades

# Verify time sync (enabled by default on Ubuntu 24.04)
timedatectl set-ntp true
timedatectl status

# Basic kernel hardening
cat > /etc/sysctl.d/99-security.conf << 'EOF'
# SYN flood protection
net.ipv4.tcp_syncookies = 1

# Reverse path filtering (anti-spoofing)
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable ICMP redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv6.conf.all.accept_redirects = 0

# Ignore ICMP broadcasts
net.ipv4.icmp_echo_ignore_broadcasts = 1
EOF

sysctl --system

Step 2: Install Tailscale

Important: Use the official repository, NOT the Snap package (has AppArmor issues with Tailscale SSH on Ubuntu 24.04).

# Add official Tailscale repository
curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.noarmor.gpg | \
  tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null
curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.tailscale-keyring.list | \
  tee /etc/apt/sources.list.d/tailscale.list

# Install
apt update
apt install -y tailscale

Step 3: Connect Tailscale with SSH

tailscale up --ssh

This outputs an authentication URL. Open it and authenticate with your Tailscale account.

Important: After connecting, go to https://login.tailscale.com/admin/machines and disable key expiry for this server to prevent lockouts.

Step 4: Configure firewalld

# Install firewalld
apt install -y firewalld

# Enable and start
systemctl enable --now firewalld

# Trust Tailscale interface (allows all traffic from your tailnet)
firewall-cmd --permanent --zone=trusted --add-interface=tailscale0

# Allow Tailscale UDP port (WireGuard)
firewall-cmd --permanent --add-port=41641/udp

# Allow web traffic (for Gordon reverse proxy)
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https

# Apply changes
firewall-cmd --reload

Note: Gordon's registry is proxied through port 80/443 via gordon_domain. Port 5000 does NOT need to be exposed publicly.

Step 5: Verify Tailscale SSH Works

From another machine on your tailnet:

# Get the server's Tailscale hostname
tailscale status

# Connect from your local machine
ssh root@<tailscale-hostname>
# or
tailscale ssh root@<tailscale-hostname>

Step 6: Block Public SSH

Only proceed once Tailscale SSH is confirmed working.

# Remove SSH from public zone
firewall-cmd --permanent --remove-service=ssh

# Set default policy to DROP all unmatched traffic
firewall-cmd --permanent --zone=public --set-target=DROP

# Apply changes
firewall-cmd --reload

Verify public SSH is blocked:

# From another machine (not on tailnet)
ssh -o ConnectTimeout=5 root@<public-ip>  # Should timeout

Step 6.5: Restrict Registry and Auth to Tailnet CIDR

If you want /auth/* and registry (/v2/*) reachable only from your tailnet, set Gordon's registry allowlist to the Tailscale CGNAT range.

Edit ~/.config/gordon/gordon.toml:

[server]
registry_allowed_ips = ["100.64.0.0/10"]

Apply the change:

systemctl --user restart gordon

Behavior:

Requests from tailnet IPs (100.64.0.0/10) are allowed.
Requests outside that CIDR get 403 Forbidden on registry/auth endpoints.
App routes still use normal host-based routing through your proxy ports.

For private operations (auth, push/pull, admin API), keep using the HTTPS registry domain, but resolve that domain to the VPS tailnet IP (for example, via Cloudflare DNS to the tailnet address). With self-signed certs, configure the CLI remote with insecure TLS enabled.

# Example: save remote once
gordon remotes add tailnet-reg https://gordon.example.com --token-env GORDON_TOKEN --insecure
gordon remotes use tailnet-reg

# Then use commands normally (auth + admin API)
gordon auth login
gordon routes list
gordon backup status

~/.config/gordon/remotes.toml should contain an entry like:

active = "tailnet-reg"

[remotes.tailnet-reg]
url = "https://gordon.example.com"
token_env = "GORDON_TOKEN"
insecure_tls = true

This keeps transport on HTTPS while allowing self-signed certificates in tailnet-only deployments.

Step 7: Install CrowdSec

# Add CrowdSec repo (NOT Ubuntu's outdated version)
curl -s https://install.crowdsec.net | bash

# Install CrowdSec
apt update
apt install -y crowdsec

Step 8: Install nftables Bouncer

Ubuntu 24.04 uses nftables by default:

apt install -y crowdsec-firewall-bouncer-nftables
systemctl enable --now crowdsec-firewall-bouncer

Step 9: Install Security Collections

# HTTP protection
cscli collections install crowdsecurity/http-cve
cscli collections install crowdsecurity/base-http-scenarios

# Reload to apply
systemctl reload crowdsec

Step 10: Whitelist Tailscale IPs

Prevent CrowdSec from blocking Tailscale traffic:

cat > /etc/crowdsec/parsers/s02-enrich/tailscale-whitelist.yaml << 'EOF'
name: tailscale-whitelist
description: "Whitelist Tailscale CGNAT range"
whitelist:
  reason: "Tailscale internal traffic"
  cidr:
    - "100.64.0.0/10"
EOF

systemctl reload crowdsec

Step 11: Enroll in CrowdSec Console

This enables community blocklists - IPs blocked across all CrowdSec users globally (15,000+ malicious IPs).

Create account at https://app.crowdsec.net
Go to Security Engines → Add Security Engine
Copy enrollment key and run:

cscli console enroll <your-enrollment-key>

Accept the enrollment in the CrowdSec console
Enable console management:

systemctl restart crowdsec
cscli console enable console_management
systemctl reload crowdsec

Verification

# Firewall status
firewall-cmd --state
firewall-cmd --list-all
firewall-cmd --list-all --zone=trusted

# CrowdSec status
cscli metrics
cscli bouncers list
cscli console status

# Verify nftables has CrowdSec rules
nft list ruleset | grep -A5 'crowdsec'

# View blocked IPs count
cscli decisions list | wc -l

Port Summary

Port	Protocol	Status	Purpose
22	TCP	Closed	SSH (use Tailscale instead)
80	TCP	Open	HTTP (Gordon proxy)
443	TCP	Open	HTTPS (Gordon proxy via Cloudflare)
5000	TCP	Closed	Registry (proxied through 80/443, containers bind to 127.0.0.1)
41641	UDP	Open	Tailscale WireGuard

Security Note: All managed containers bind to 127.0.0.1 by default, preventing direct access to their ports. The only entry point to your applications is Gordon's reverse proxy on ports 80/443, where authentication and rate limiting are enforced.

Troubleshooting

Can't SSH via Tailscale

Check for ACL issues:

journalctl -u tailscaled --since "10 minutes ago" | grep -i "rejected\|acl"

If you see rejected due to acl, check your Tailscale ACLs at https://login.tailscale.com/admin/acls

Locked out completely

Use your VPS provider's console access (e.g., Hetzner Cloud Console):

firewall-cmd --add-service=ssh  # Temporarily re-enable public SSH

CrowdSec not blocking

Verify bouncer is running:

systemctl status crowdsec-firewall-bouncer
cscli bouncers list  # Should show "Valid"

Check what's being blocked

cscli decisions list
cscli alerts list

Next Steps

Install Gordon
Podman Rootless Setup - Enhanced container isolation
Cloudflare DNS & Proxy Setup - HTTPS termination (coming soon)

References

source

1# Secure VPS Setup
2
3A production-ready VPS security configuration with Tailscale-only SSH access and CrowdSec intrusion prevention.
4
5## Overview
6
7This guide configures three security layers:
8
9| Layer | Protection |
10|-------|------------|
11| SSH | Tailscale only (invisible to internet) |
12| Firewall | firewalld with DROP policy |
13| IPS | CrowdSec with community blocklists |
14
15## Prerequisites
16
17- Fresh Ubuntu 24.04 VPS
18- Tailscale account
19- Initial SSH access via public IP (temporary)
20- Cloudflare account for HTTPS termination
21
22---
23
24## Step 1: Initial System Hardening
25
26Before anything else, enable automatic security updates and time sync:
27
28```bash
29# Automatic security updates
30apt update && apt install -y unattended-upgrades
31dpkg-reconfigure -plow unattended-upgrades
32
33# Verify time sync (enabled by default on Ubuntu 24.04)
34timedatectl set-ntp true
35timedatectl status
36
37# Basic kernel hardening
38cat > /etc/sysctl.d/99-security.conf << 'EOF'
39# SYN flood protection
40net.ipv4.tcp_syncookies = 1
41
42# Reverse path filtering (anti-spoofing)
43net.ipv4.conf.all.rp_filter = 1
44net.ipv4.conf.default.rp_filter = 1
45
46# Disable ICMP redirects
47net.ipv4.conf.all.accept_redirects = 0
48net.ipv4.conf.all.send_redirects = 0
49net.ipv6.conf.all.accept_redirects = 0
50
51# Ignore ICMP broadcasts
52net.ipv4.icmp_echo_ignore_broadcasts = 1
53EOF
54
55sysctl --system
56```
57
58## Step 2: Install Tailscale
59
60> **Important**: Use the official repository, NOT the Snap package (has AppArmor issues with Tailscale SSH on Ubuntu 24.04).
61
62```bash
63# Add official Tailscale repository
64curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.noarmor.gpg | \
65  tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null
66curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.tailscale-keyring.list | \
67  tee /etc/apt/sources.list.d/tailscale.list
68
69# Install
70apt update
71apt install -y tailscale
72```
73
74## Step 3: Connect Tailscale with SSH
75
76```bash
77tailscale up --ssh
78```
79
80This outputs an authentication URL. Open it and authenticate with your Tailscale account.
81
82**Important**: After connecting, go to https://login.tailscale.com/admin/machines and **disable key expiry** for this server to prevent lockouts.
83
84## Step 4: Configure firewalld
85
86```bash
87# Install firewalld
88apt install -y firewalld
89
90# Enable and start
91systemctl enable --now firewalld
92
93# Trust Tailscale interface (allows all traffic from your tailnet)
94firewall-cmd --permanent --zone=trusted --add-interface=tailscale0
95
96# Allow Tailscale UDP port (WireGuard)
97firewall-cmd --permanent --add-port=41641/udp
98
99# Allow web traffic (for Gordon reverse proxy)
100firewall-cmd --permanent --add-service=http
101firewall-cmd --permanent --add-service=https
102
103# Apply changes
104firewall-cmd --reload
105```
106
107> **Note**: Gordon's registry is proxied through port 80/443 via `gordon_domain`. Port 5000 does NOT need to be exposed publicly.
108
109## Step 5: Verify Tailscale SSH Works
110
111From another machine on your tailnet:
112
113```bash
114# Get the server's Tailscale hostname
115tailscale status
116
117# Connect from your local machine
118ssh root@<tailscale-hostname>
119# or
120tailscale ssh root@<tailscale-hostname>
121```
122
123## Step 6: Block Public SSH
124
125**Only proceed once Tailscale SSH is confirmed working.**
126
127```bash
128# Remove SSH from public zone
129firewall-cmd --permanent --remove-service=ssh
130
131# Set default policy to DROP all unmatched traffic
132firewall-cmd --permanent --zone=public --set-target=DROP
133
134# Apply changes
135firewall-cmd --reload
136```
137
138Verify public SSH is blocked:
139
140```bash
141# From another machine (not on tailnet)
142ssh -o ConnectTimeout=5 root@<public-ip>  # Should timeout
143```
144
145## Step 6.5: Restrict Registry and Auth to Tailnet CIDR
146
147If you want `/auth/*` and registry (`/v2/*`) reachable only from your tailnet, set Gordon's registry allowlist to the Tailscale CGNAT range.
148
149Edit `~/.config/gordon/gordon.toml`:
150
151```toml
152[server]
153registry_allowed_ips = ["100.64.0.0/10"]
154```
155
156Apply the change:
157
158```bash
159systemctl --user restart gordon
160```
161
162Behavior:
163
164- Requests from tailnet IPs (`100.64.0.0/10`) are allowed.
165- Requests outside that CIDR get `403 Forbidden` on registry/auth endpoints.
166- App routes still use normal host-based routing through your proxy ports.
167
168For private operations (auth, push/pull, admin API), keep using the HTTPS registry domain, but resolve that domain to the VPS tailnet IP (for example, via Cloudflare DNS to the tailnet address). With self-signed certs, configure the CLI remote with insecure TLS enabled.
169
170```bash
171# Example: save remote once
172gordon remotes add tailnet-reg https://gordon.example.com --token-env GORDON_TOKEN --insecure
173gordon remotes use tailnet-reg
174
175# Then use commands normally (auth + admin API)
176gordon auth login
177gordon routes list
178gordon backup status
179```
180
181`~/.config/gordon/remotes.toml` should contain an entry like:
182
183```toml
184active = "tailnet-reg"
185
186[remotes.tailnet-reg]
187url = "https://gordon.example.com"
188token_env = "GORDON_TOKEN"
189insecure_tls = true
190```
191
192This keeps transport on HTTPS while allowing self-signed certificates in tailnet-only deployments.
193
194---
195
196## Step 7: Install CrowdSec
197
198```bash
199# Add CrowdSec repo (NOT Ubuntu's outdated version)
200curl -s https://install.crowdsec.net | bash
201
202# Install CrowdSec
203apt update
204apt install -y crowdsec
205```
206
207## Step 8: Install nftables Bouncer
208
209Ubuntu 24.04 uses nftables by default:
210
211```bash
212apt install -y crowdsec-firewall-bouncer-nftables
213systemctl enable --now crowdsec-firewall-bouncer
214```
215
216## Step 9: Install Security Collections
217
218```bash
219# HTTP protection
220cscli collections install crowdsecurity/http-cve
221cscli collections install crowdsecurity/base-http-scenarios
222
223# Reload to apply
224systemctl reload crowdsec
225```
226
227## Step 10: Whitelist Tailscale IPs
228
229Prevent CrowdSec from blocking Tailscale traffic:
230
231```bash
232cat > /etc/crowdsec/parsers/s02-enrich/tailscale-whitelist.yaml << 'EOF'
233name: tailscale-whitelist
234description: "Whitelist Tailscale CGNAT range"
235whitelist:
236  reason: "Tailscale internal traffic"
237  cidr:
238    - "100.64.0.0/10"
239EOF
240
241systemctl reload crowdsec
242```
243
244## Step 11: Enroll in CrowdSec Console
245
246This enables community blocklists - IPs blocked across all CrowdSec users globally (15,000+ malicious IPs).
247
2481. Create account at https://app.crowdsec.net
2492. Go to **Security Engines** → **Add Security Engine**
2503. Copy enrollment key and run:
251
252```bash
253cscli console enroll <your-enrollment-key>
254```
255
2564. Accept the enrollment in the CrowdSec console
2575. Enable console management:
258
259```bash
260systemctl restart crowdsec
261cscli console enable console_management
262systemctl reload crowdsec
263```
264
265---
266
267## Verification
268
269```bash
270# Firewall status
271firewall-cmd --state
272firewall-cmd --list-all
273firewall-cmd --list-all --zone=trusted
274
275# CrowdSec status
276cscli metrics
277cscli bouncers list
278cscli console status
279
280# Verify nftables has CrowdSec rules
281nft list ruleset | grep -A5 'crowdsec'
282
283# View blocked IPs count
284cscli decisions list | wc -l
285```
286
287---
288
289## Port Summary
290
291| Port | Protocol | Status | Purpose |
292|------|----------|--------|---------|
293| 22 | TCP | Closed | SSH (use Tailscale instead) |
294| 80 | TCP | Open | HTTP (Gordon proxy) |
295| 443 | TCP | Open | HTTPS (Gordon proxy via Cloudflare) |
296| 5000 | TCP | Closed | Registry (proxied through 80/443, containers bind to 127.0.0.1) |
297| 41641 | UDP | Open | Tailscale WireGuard |
298
299> **Security Note:** All managed containers bind to `127.0.0.1` by default, preventing direct access to their ports. The only entry point to your applications is Gordon's reverse proxy on ports 80/443, where authentication and rate limiting are enforced.
300
301---
302
303## Troubleshooting
304
305### Can't SSH via Tailscale
306
307Check for ACL issues:
308
309```bash
310journalctl -u tailscaled --since "10 minutes ago" | grep -i "rejected\|acl"
311```
312
313If you see `rejected due to acl`, check your Tailscale ACLs at https://login.tailscale.com/admin/acls
314
315### Locked out completely
316
317Use your VPS provider's console access (e.g., Hetzner Cloud Console):
318
319```bash
320firewall-cmd --add-service=ssh  # Temporarily re-enable public SSH
321```
322
323### CrowdSec not blocking
324
325Verify bouncer is running:
326
327```bash
328systemctl status crowdsec-firewall-bouncer
329cscli bouncers list  # Should show "Valid"
330```
331
332### Check what's being blocked
333
334```bash
335cscli decisions list
336cscli alerts list
337```
338
339---
340
341## Next Steps
342
343- [Install Gordon](/docs/getting-started.md)
344- [Podman Rootless Setup](./podman-rootless.md) - Enhanced container isolation
345- [Cloudflare DNS & Proxy Setup](#) - HTTPS termination (coming soon)
346
347## References
348
349- [Tailscale SSH Documentation](https://tailscale.com/kb/1193/tailscale-ssh)
350- [firewalld Documentation](https://firewalld.org/documentation/)
351- [CrowdSec Documentation](https://docs.crowdsec.net/)
352- [CrowdSec Console](https://app.crowdsec.net)

secure-vps-setup.md RawHistoryFolder