140 lines
8.0 KiB
Markdown
140 lines
8.0 KiB
Markdown
# Tailscale Mesh Connectivity Test
|
|
|
|
Last tested: 2026-03-31
|
|
|
|
## Test Method
|
|
|
|
Full `tailscale ping` from each SSH-accessible host to all other online Headscale nodes. This tests WireGuard tunnel establishment and reports whether the connection is direct (peer-to-peer) or relayed via DERP.
|
|
|
|
## Results
|
|
|
|
All 10 online hosts can reach all other online hosts. No failures.
|
|
|
|
### Connection Type Matrix
|
|
|
|
`D` = direct, `R` = DERP relay, `—` = self
|
|
|
|
| From \ To | Atlantis | Calypso | Pi-5 | Homelab | Matrix-Ubuntu | Setillo | NUC | Guava | Seattle | PVE |
|
|
|-----------|----------|---------|------|---------|---------------|---------|-----|-------|---------|-----|
|
|
| **Atlantis** | — | D | D | D | D | D | D | D | D | D |
|
|
| **Calypso** | D | — | D | D | D | R | D | D | D | D |
|
|
| **Pi-5** | D | D | — | D | D | D | D | D | D | D |
|
|
| **Homelab-VM** | D | D | D | — | D | R | D | D | D | D |
|
|
| **Matrix-Ubuntu** | (tested inbound) | | | | — | | | | | |
|
|
| **Setillo** | D | D | D | R | R | — | | | | |
|
|
| **NUC** | D | D | D | D | R | D | — | R | D | R |
|
|
| **Guava** | (no CLI) | | | | | | | — | | |
|
|
| **Seattle** | D | D | D | D | D | D | R | D | — | D |
|
|
| **PVE** | D | D | D | D | D | D | D | D | D | — |
|
|
|
|
### Notes
|
|
|
|
- **Atlantis/Calypso**: Tailscale binary at `/var/packages/Tailscale/target/bin/tailscale` (Synology package)
|
|
- **Setillo**: Tailscale binary at `/usr/local/bin/tailscale`
|
|
- **Guava**: Tailscale runs via TrueNAS built-in management; no `tailscale` CLI in PATH. Confirmed reachable via inbound pings from all other hosts.
|
|
- **Matrix-Ubuntu**: SSH via LAN IP (192.168.0.154) was unreliable during testing due to table 52 LAN interception (since fixed). Confirmed reachable via Tailscale IP and inbound pings.
|
|
- **DERP relays** (NUC ↔ some peers, Setillo ↔ homelab/matrix-ubuntu): Normal for nodes behind different NATs. Adds 15-60ms latency but does not affect reliability. Connections may upgrade to direct over time.
|
|
|
|
### ICMP Ping Notes
|
|
|
|
Standard ICMP ping from the OS (not `tailscale ping`) fails for Atlantis, Calypso, and Setillo because those hosts have ICMP blocked at the firewall level. This does not indicate a connectivity problem — `tailscale ping` and SSH both work.
|
|
|
|
## Tailscale Health Warnings
|
|
|
|
After fixes applied on 2026-03-31:
|
|
|
|
| Host | Health |
|
|
|------|--------|
|
|
| Homelab-VM | none |
|
|
| Pi-5 | none |
|
|
| NUC | none |
|
|
| Seattle | none |
|
|
| PVE | none (was `--accept-routes is false`, fixed) |
|
|
| Matrix-Ubuntu | none (was `--accept-routes is false`, fixed) |
|
|
|
|
## Fixes Applied (2026-03-31)
|
|
|
|
### LAN Routing (table 52 interception)
|
|
|
|
1. **Pi-5**: Enabled `--accept-routes`, added LAN routing rule (priority 5200), persistent via dispatcher script + cron
|
|
2. **Matrix-Ubuntu**: Enabled `--accept-routes`, added LAN routing rule (priority 5200), persistent via dispatcher script + cron. Enabling `--accept-routes` without the rule broke LAN connectivity (SSH via 192.168.0.154 timed out).
|
|
3. **PVE**: Enabled `--accept-routes`, added LAN routing rule (priority 5200), persistent via cron @reboot
|
|
|
|
See [LAN Routing Fix](GUAVA_LAN_ROUTING_FIX.md) for full details on the table 52 issue.
|
|
|
|
### Kuma monitors
|
|
|
|
- Switched **all 20 Calypso monitors** from Tailscale IP (`100.103.48.78`) to LAN IP (`192.168.0.250`) in the Kuma SQLite DB. Pi-5 (where Kuma runs) is on the same LAN, so using Tailscale IPs added unnecessary fragility.
|
|
- Added LAN-based monitors for Rustdesk (ID:124) and Rackula (ID:125).
|
|
- Fixed corrupted `accepted_statuscodes_json` field (`[200-299]` → `["200-299"]`) that prevented the Kuma UI from loading.
|
|
- Fixed ntfy notifications by setting `primaryBaseURL` to `https://kuma.vish.gg` — the "view" action button was missing a URL.
|
|
|
|
### Calypso Tailscale 5-minute disconnect (root cause)
|
|
|
|
**Symptom**: Calypso's disco key rotated every 5 minutes, dropping all peer WireGuard sessions for ~30 seconds.
|
|
|
|
**Root cause**: A cron job in `/etc/crontab` ran `/usr/local/bin/tailscale-fix.sh` every 5 minutes (`*/5 * * * *`). The script checked for the `tailscale0` TUN device, but Calypso runs Tailscale in `--tun=userspace-networking` mode (Synology DSM7), which has no TUN device. The script also checked `tailscale status --json` which returned empty state when run as the `tailscale` user. So every 5 minutes:
|
|
|
|
1. Script detects "tailscale0 missing" or "state empty"
|
|
2. Runs `configure-host` + full service restart via `synosystemctl`
|
|
3. Re-authenticates with `tailscale up --reset`
|
|
4. New disco key generated → all peers tear down and re-establish connections
|
|
|
|
**Fix**: Rewrote `/usr/local/bin/tailscale-fix.sh` to check if `tailscaled` process is running and can `tailscale ping` a known peer (Atlantis), which works in both TUN and userspace-networking modes.
|
|
|
|
**Additional changes on Calypso** (not the root cause but good hygiene):
|
|
- Disabled Docker IPv6 on all bridge interfaces via `sysctl` (77 routes removed)
|
|
- Updated `dockerd.json` with `"ipv6": false, "ip6tables": false` (persistent after Docker restart)
|
|
- Added `TS_DEBUG_NETMON_SKIP_INTERFACE_REGEXPS` and `TS_DEBUG_NETMON_NO_ROUTE_MONITORING` env vars to Tailscale startup script
|
|
- Added `/etc/hosts` entry: `192.168.0.250 headscale.vish.gg` to avoid hairpin NAT for control plane
|
|
|
|
### GL.iNet router Tailscale fixes
|
|
|
|
**GL-MT3000** (Hawaii, via Jellyfish at `192.168.12.1`):
|
|
- Tailscale was pointed at `https://controlplane.tailscale.com` (official) instead of Headscale
|
|
- Re-registered with preauth key: `tailscale up --login-server=https://headscale.vish.gg:8443 --authkey=<key> --hostname=gl-mt3000 --accept-routes --accept-dns=false --advertise-routes=192.168.12.0/24 --reset`
|
|
- Auto-start verified: `/etc/init.d/tailscale-up` (S81) runs `tailscale up` with correct flags after 10s delay
|
|
- State file at `/etc/tailscale/tailscaled.state` has `WantRunning: true`, `ControlURL: headscale.vish.gg:8443`
|
|
|
|
**GL-BE3600** (Home gateway at `192.168.0.1`):
|
|
- Was already connected to Headscale correctly, no fix needed
|
|
- SSH not available on LAN port 22 — accessible via Tailscale IP (`100.105.59.123`)
|
|
- Auto-start verified: `/etc/init.d/tailscale-up` (S99) waits up to 120s for network, then connects with `--advertise-routes=192.168.68.0/22,192.168.8.0/24`
|
|
|
|
### Speedtest results (Ookla official CLI, staggered)
|
|
|
|
Tests run sequentially per WAN to avoid bandwidth contention. Tested 2026-03-31.
|
|
|
|
| Host | Location | NIC | ISP | Download | Upload | Latency | Loss |
|
|
|------|----------|-----|-----|----------|--------|---------|------|
|
|
| Calypso | Home | 10GbE | Sonic | 8,080 Mbps | 6,847 Mbps | 3 ms | 0% |
|
|
| Atlantis | Home | 10GbE | Sonic | 6,079 Mbps | 6,908 Mbps | 3 ms | 0% |
|
|
| Matrix-Ubuntu | Home | virtio (on Atlantis) | Sonic | 3,537 Mbps | 3,666 Mbps | 4 ms | 0% |
|
|
| GL-MT3000 | Hawaii | 1GbE | Spectrum | 1,051 Mbps | 37 Mbps | 8 ms | 0% |
|
|
| NUC | Concord | 1GbE | Comcast | 943 Mbps | 357 Mbps | 14 ms | 3.7% |
|
|
| GL-BE3600 | Home (gateway) | 2.5GbE | Comcast | 781 Mbps | 71 Mbps | 25 ms | 0% |
|
|
| Homelab-VM | Home | 1GbE virtio | Sonic | 616 Mbps | 933 Mbps | 3 ms | 0% |
|
|
| Pi-5 | Home | 1GbE | Sonic | 551 Mbps | 936 Mbps | 3 ms | 0% |
|
|
| Seattle VPS | Seattle | VPS | Nubes LLC | 197 Mbps | 197 Mbps | 1 ms | 0% |
|
|
| Setillo | Remote | Starlink | Starlink | 72 Mbps | 27 Mbps | 27 ms | 0% |
|
|
|
|
### iperf3 benchmarks (all hosts → Calypso)
|
|
|
|
All tests against Calypso's iperf3 server (`192.168.0.250:5201` for LAN, `100.103.48.78:5201` for Tailscale). Run staggered to avoid contention. Tested 2026-03-31.
|
|
|
|
| Source | Link | NIC / Tunnel | Streams | Throughput | Retransmits |
|
|
|--------|------|-------------|---------|------------|-------------|
|
|
| Atlantis | LAN | 10GbE | 4 | 9.30 Gbps | 0 |
|
|
| Guava | LAN | 10GbE | 4 | 8.54 Gbps | 24 |
|
|
| Homelab-VM | LAN | 1GbE virtio | 1 | 940 Mbps | 0 |
|
|
| PVE | LAN | 1GbE | 1 | 938 Mbps | 0 |
|
|
| Pi-5 | LAN | 1GbE | 1 | 887 Mbps | 451 |
|
|
| Seattle | Tailscale | WireGuard | 1 | 454 Mbps | 410 |
|
|
| NUC | Tailscale | WireGuard | 1 | 252 Mbps | 2 |
|
|
|
|
Notes:
|
|
- 10GbE hosts (Atlantis, Guava) saturate REDACTED_APP_PASSWORD 4 parallel streams
|
|
- 1GbE hosts cap at ~940 Mbps as expected; Pi-5 slightly lower with retransmits
|
|
- Tailscale/WireGuard tunnel adds overhead: Seattle gets ~450 Mbps, NUC ~250 Mbps
|
|
- Calypso not tested (it's the server)
|