homelab-optimized/docs/networking/TAILSCALE_MESH_TEST.md

Tailscale Mesh Connectivity Test

Last tested: 2026-03-31

Test Method

Full tailscale ping from each SSH-accessible host to all other online Headscale nodes. This tests WireGuard tunnel establishment and reports whether the connection is direct (peer-to-peer) or relayed via DERP.

Results

All 10 online hosts can reach all other online hosts. No failures.

Connection Type Matrix

D = direct, R = DERP relay, — = self

From \ To	Atlantis	Calypso	Pi-5	Homelab	Matrix-Ubuntu	Setillo	NUC	Guava	Seattle	PVE
Atlantis	—	D	D	D	D	D	D	D	D	D
Calypso	D	—	D	D	D	R	D	D	D	D
Pi-5	D	D	—	D	D	D	D	D	D	D
Homelab-VM	D	D	D	—	D	R	D	D	D	D
Matrix-Ubuntu	(tested inbound)				—
Setillo	D	D	D	R	R	—
NUC	D	D	D	D	R	D	—	R	D	R
Guava	(no CLI)							—
Seattle	D	D	D	D	D	D	R	D	—	D
PVE	D	D	D	D	D	D	D	D	D	—

Notes

• Atlantis/Calypso: Tailscale binary at /var/packages/Tailscale/target/bin/tailscale (Synology package)
• Setillo: Tailscale binary at /usr/local/bin/tailscale
• Guava: Tailscale runs via TrueNAS built-in management; no tailscale CLI in PATH. Confirmed reachable via inbound pings from all other hosts.
• Matrix-Ubuntu: SSH via LAN IP (192.168.0.154) was unreliable during testing due to table 52 LAN interception (since fixed). Confirmed reachable via Tailscale IP and inbound pings.
• DERP relays (NUC ↔ some peers, Setillo ↔ homelab/matrix-ubuntu): Normal for nodes behind different NATs. Adds 15-60ms latency but does not affect reliability. Connections may upgrade to direct over time.

ICMP Ping Notes

Standard ICMP ping from the OS (not tailscale ping) fails for Atlantis, Calypso, and Setillo because those hosts have ICMP blocked at the firewall level. This does not indicate a connectivity problem — tailscale ping and SSH both work.

Tailscale Health Warnings

After fixes applied on 2026-03-31:

Host	Health
Homelab-VM	none
Pi-5	none
NUC	none
Seattle	none
PVE	none (was --accept-routes is false, fixed)
Matrix-Ubuntu	none (was --accept-routes is false, fixed)

Fixes Applied (2026-03-31)

LAN Routing (table 52 interception)

1. Pi-5: Enabled --accept-routes, added LAN routing rule (priority 5200), persistent via dispatcher script + cron
2. Matrix-Ubuntu: Enabled --accept-routes, added LAN routing rule (priority 5200), persistent via dispatcher script + cron. Enabling --accept-routes without the rule broke LAN connectivity (SSH via 192.168.0.154 timed out).
3. PVE: Enabled --accept-routes, added LAN routing rule (priority 5200), persistent via cron @reboot

See LAN Routing Fix for full details on the table 52 issue.

Kuma monitors

• Switched all 20 Calypso monitors from Tailscale IP (100.103.48.78) to LAN IP (192.168.0.250) in the Kuma SQLite DB. Pi-5 (where Kuma runs) is on the same LAN, so using Tailscale IPs added unnecessary fragility.
• Added LAN-based monitors for Rustdesk (ID:124) and Rackula (ID:125).
• Fixed corrupted accepted_statuscodes_json field ([200-299] → ["200-299"]) that prevented the Kuma UI from loading.
• Fixed ntfy notifications by setting primaryBaseURL to https://kuma.vish.gg — the "view" action button was missing a URL.

Calypso Tailscale 5-minute disconnect (root cause)

Symptom: Calypso's disco key rotated every 5 minutes, dropping all peer WireGuard sessions for ~30 seconds.

Root cause: A cron job in /etc/crontab ran /usr/local/bin/tailscale-fix.sh every 5 minutes (*/5 * * * *). The script checked for the tailscale0 TUN device, but Calypso runs Tailscale in --tun=userspace-networking mode (Synology DSM7), which has no TUN device. The script also checked tailscale status --json which returned empty state when run as the tailscale user. So every 5 minutes:

1. Script detects "tailscale0 missing" or "state empty"
2. Runs configure-host + full service restart via synosystemctl
3. Re-authenticates with tailscale up --reset
4. New disco key generated → all peers tear down and re-establish connections

Fix: Rewrote /usr/local/bin/tailscale-fix.sh to check if tailscaled process is running and can tailscale ping a known peer (Atlantis), which works in both TUN and userspace-networking modes.

Additional changes on Calypso (not the root cause but good hygiene):

• Disabled Docker IPv6 on all bridge interfaces via sysctl (77 routes removed)
• Updated dockerd.json with "ipv6": false, "ip6tables": false (persistent after Docker restart)
• Added TS_DEBUG_NETMON_SKIP_INTERFACE_REGEXPS and TS_DEBUG_NETMON_NO_ROUTE_MONITORING env vars to Tailscale startup script
• Added /etc/hosts entry: 192.168.0.250 headscale.vish.gg to avoid hairpin NAT for control plane

GL.iNet router Tailscale fixes

GL-MT3000 (Hawaii, via Jellyfish at 192.168.12.1):

• Tailscale was pointed at https://controlplane.tailscale.com (official) instead of Headscale
• Re-registered with preauth key: tailscale up --login-server=https://headscale.vish.gg:8443 --authkey=<key> --hostname=gl-mt3000 --accept-routes --accept-dns=false --advertise-routes=192.168.12.0/24 --reset
• Auto-start verified: /etc/init.d/tailscale-up (S81) runs tailscale up with correct flags after 10s delay
• State file at /etc/tailscale/tailscaled.state has WantRunning: true, ControlURL: headscale.vish.gg:8443

GL-BE3600 (Home gateway at 192.168.0.1):

• Was already connected to Headscale correctly, no fix needed
• SSH not available on LAN port 22 — accessible via Tailscale IP (100.105.59.123)
• Auto-start verified: /etc/init.d/tailscale-up (S99) waits up to 120s for network, then connects with --advertise-routes=192.168.68.0/22,192.168.8.0/24

Speedtest results (Ookla official CLI)

Host	Location	NIC	ISP	Download	Upload	Latency
Calypso	Home	10GbE	Sonic	6,255 Mbps	7,039 Mbps	2 ms
Atlantis	Home	10GbE	Sonic	4,623 Mbps	6,988 Mbps	3 ms
Matrix-Ubuntu	Home	virtio	Sonic	1,907 Mbps	6,330 Mbps	3 ms
GL-MT3000	Hawaii	1GbE	Spectrum	1,127 Mbps	35 Mbps	10 ms
NUC	Concord	1GbE	Comcast	882 Mbps	361 Mbps	13 ms
Pi-5	Home	1GbE	Sonic	710 Mbps	936 Mbps	2 ms
Homelab-VM	Home	1GbE virtio	Sonic	448 Mbps	931 Mbps	3 ms

Internal 10GbE link (Atlantis ↔ Calypso via iperf3): 8.61 Gbps with 4 parallel streams, 0 retransmits.