Vish/homelab-optimized
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
abd959b47aad21e4a0fca994c66f0b4142f441c9
homelab-optimized/docs/infrastructure/headscale-migration-guide.md
Gitea Mirror Bot abd959b47a
Some checks failed
Documentation / Deploy to GitHub Pages (push) Has been cancelled
Details
Documentation / Build Docusaurus (push) Has been cancelled
Details
Sanitized mirror from private repository - 2026-04-05 05:12:55 UTC
2026-04-05 05:12:55 +00:00

16 KiB
Raw Blame History

Headscale Migration Guide

Overview

This homelab uses a self-hosted Headscale instance instead of Tailscale cloud. Headscale is a drop-in open-source replacement for the Tailscale control server.

  • Headscale server: https://headscale.vish.gg:8443
  • MagicDNS suffix: tail.vish.gg (e.g. atlantis.tail.vish.gg)
  • Login: Authentik SSO at sso.vish.gg — username vish or email admin@thevish.io
  • Hosted on: Calypso (192.168.0.250), managed via Docker

Connecting a New Device

Linux (Ubuntu / Debian)

  1. Install Tailscale if not already installed:

    curl -fsSL https://tailscale.com/install.sh | sh

  2. Connect to headscale:

    sudo tailscale up \
  --login-server=https://headscale.vish.gg:8443 \
  --accept-routes \
  --force-reauth

  3. A browser auth URL will be printed. Open it and log in with Authentik SSO.

  4. If DNS doesn't resolve headscale.vish.gg (e.g. fresh machine with no AdGuard), add a temporary hosts entry first:

    echo '184.23.52.14 headscale.vish.gg' | sudo tee -a /etc/hosts
# Run tailscale up, then clean up:
sudo sed -i '/headscale.vish.gg/d' /etc/hosts

  5. If the machine was previously on Tailscale cloud and complains about non-default flags, Tailscale will print the exact command with all required flags — copy and run that command.

Note

: After registration, an admin must approve the node and fix the IP if preserving the original Tailscale IP (see Admin section below).

Windows

  1. Download and install Tailscale from https://tailscale.com/download/windows

  2. Open PowerShell as Administrator and run:

    tailscale up --login-server=https://headscale.vish.gg:8443 --accept-routes --force-reauth

  3. A browser window will open — log in with Authentik SSO (vish / admin@thevish.io).

  4. If it shows a "mention all non-default flags" error, copy and run the exact command it provides, adding --login-server=https://headscale.vish.gg:8443 --force-reauth to it.

Important

: Always include --accept-routes on Windows otherwise subnet routes (e.g. 192.168.0.x) won't be reachable.

iOS (iPhone / iPad)

  1. Install Tailscale from the App Store.

  2. Open the app → tap your account icon (top right) → Log in

  3. Tap the ··· menu (top right of the login screen) → Use custom coordination server

  4. Enter: https://headscale.vish.gg:8443 → Save

  5. Log in with Authentik SSO — username vish or email admin@thevish.io

Note

: .vish.local hostnames do NOT work on iOS — iOS intercepts .local for mDNS and never forwards to DNS. Use Tailscale IPs (100.x.x.x) or MagicDNS names (hostname.tail.vish.gg) instead.

macOS

  1. Install Tailscale from the App Store or https://tailscale.com/download/mac

  2. Option A — GUI: Click the Tailscale menu bar icon → Preferences → hold Option while clicking "Log in" to enter a custom server URL → enter https://headscale.vish.gg:8443

  3. Option B — CLI:

    sudo tailscale up \
  --login-server=https://headscale.vish.gg:8443 \
  --accept-routes \
  --force-reauth

  4. Log in with Authentik SSO when the browser opens.

Note

: Same as iOS, .vish.local hostnames won't resolve on macOS when remote. Use hostname.tail.vish.gg or the Tailscale IP instead.

GL.iNet Routers (OpenWrt)

  1. SSH into the router.

  2. Add a hosts entry (since GL routers don't use AdGuard):

    echo '184.23.52.14 headscale.vish.gg' >> /etc/hosts

  3. Run tailscale up — it will error with the required flags. Copy and run the exact command it provides, appending:

    --login-server=https://headscale.vish.gg:8443 --auth-key=<preauth-key> --force-reauth

    Get a pre-auth key from an admin (see below).

  4. If advertising subnet routes, add --advertise-routes=<subnet> to the command.

Home Assistant (Tailscale Add-on)

Note

: HA Green does not expose SSH by default. Use the WebSocket API approach below, which works fully remotely via a Tailscale-connected hop host.

Remote migration steps (no physical access required):

  1. Reach HA via a hop host on the same LAN (e.g. jellyfish at 100.69.121.120):

    ssh lulu@100.69.121.120
curl http://192.168.12.202:8123/api/  # confirm HA reachable

  2. If the add-on was previously authenticated to Tailscale cloud, it will refuse --login-server change with: can't change --login-server without --force-reauth. Fix: uninstall + reinstall the add-on via supervisor API to clear tailscaled.state:

    # Via HA WebSocket API (supervisor/api endpoint):
{"type": "supervisor/api", "endpoint": "/addons/a0d7b954_tailscale/uninstall", "method": "post"}
{"type": "supervisor/api", "endpoint": "/addons/a0d7b954_tailscale/install", "method": "post"}

  3. Set options before starting:

    {"type": "supervisor/api", "endpoint": "/addons/a0d7b954_tailscale/options", "method": "post",
 "data": {"options": {"login_server": "https://headscale.vish.gg:8443", "accept_dns": false}}}

  4. Start the add-on via hassio/addon_start service, then read logs:

    GET http://192.168.12.202:8123/api/hassio/addons/a0d7b954_tailscale/logs

    Look for: AuthURL is https://headscale.vish.gg:8443/register/<key>

  5. Register on Calypso:

    docker exec headscale headscale nodes register --user vish --key <key-from-log>

  6. Fix IP via SQLite (see section above) and restart headscale.

Admin: Registering a New Node

After a node connects, an admin needs to:

1. Generate a Pre-Auth Key (optional, avoids browser auth)

ssh -p 62000 Vish@192.168.0.250
sudo /volume1/@appstore/REDACTED_APP_PASSWORD/usr/bin/docker exec headscale \
  headscale preauthkeys create --user 1 --expiration 1h

Use --authkey=<key> instead of browser auth in tailscale up.

2. Check Registered Nodes

sudo /volume1/@appstore/REDACTED_APP_PASSWORD/usr/bin/docker exec headscale headscale nodes list

3. Preserve Original Tailscale IP (if migrating from Tailscale cloud)

Headscale v0.28+ removed the --ipv4 flag. Fix IPs via SQLite:

sudo sqlite3 /volume1/@docker/volumes/headscale-data/_data/db.sqlite \
  "UPDATE nodes SET ipv4='100.x.x.x' WHERE id=<node-id>;"
sudo /volume1/@appstore/REDACTED_APP_PASSWORD/usr/bin/docker restart headscale

4. Rename a Node

sudo /volume1/@appstore/REDACTED_APP_PASSWORD/usr/bin/docker exec headscale \
  headscale nodes rename -i <id> <new-name>

5. Approve Subnet Routes

Routes advertised by nodes must be explicitly approved:

sudo /volume1/@appstore/REDACTED_APP_PASSWORD/usr/bin/docker exec headscale \
  headscale nodes approve-routes -i <node-id> -r <subnet>
# e.g. -r 192.168.0.0/24

Check all routes (v0.28 — routes are embedded in node JSON output):

sudo /volume1/@appstore/REDACTED_APP_PASSWORD/usr/bin/docker exec headscale \
  headscale nodes list --output json | python3 -c "
import sys,json
for n in json.load(sys.stdin):
    r=n.get('available_routes',[])
    a=n.get('approved_routes',[])
    if r: print(n['given_name'], 'available:', r, 'approved:', a)
"

DNS Notes

  • MagicDNS: Headscale pushes 192.168.0.250 (Calypso AdGuard) as DNS to all tailnet clients
  • AdGuard rewrites: *.vish.local names resolve to their Tailscale IPs via AdGuard rewrites on Calypso
  • .vish.local on iOS/macOS: Does NOT work remotely — iOS/macOS intercept .local for mDNS. Use hostname.tail.vish.gg instead
  • External DNS: headscale.vish.gg resolves to 184.23.52.14 (home WAN) externally, 192.168.0.250 internally via AdGuard rewrite

Uptime Kuma Monitoring

Kuma runs on pi-5 (100.77.151.40) inside the uptime-kuma container. DB at /app/data/kuma.db.

Monitor groups and hosts

Group Host Tailscale IP
Homelab homelab.tail.vish.gg 100.67.40.126
Atlantis atlantis.tail.vish.gg 100.83.230.112
Calypso calypso.tail.vish.gg 100.103.48.78
Concord_NUC vish-concord-nuc.tail.vish.gg 100.72.55.21
Setillo setillo.tail.vish.gg 100.125.0.20
Proxmox_NUC pve.tail.vish.gg 100.87.12.28
Guava truenas-scale.tail.vish.gg 100.75.252.64
Seattle seattle.tail.vish.gg 100.82.197.124
Raspberry Pi 5 100.77.151.40 100.77.151.40

Firewall rules required for Kuma (pi-5 = 100.77.151.40)

Kuma polls via Tailscale IP. Each host with a ts-input/ts-forward chain needs ACCEPT rules for pi-5:

  • Homelab VM: Rules in iptables-legacy ts-input/ts-forward for pi-5 on all monitored ports. Persisted via netfilter-persistent.
  • Concord NUC: Same — ts-input/ts-forward ACCEPT for pi-5 on monitored ports.
  • Seattle: UFW rule ufw allow from 100.77.151.40 to any port 8444
  • Calypso/Atlantis/Setillo: No ts-input blocking — Tailscale is in userspace mode on Synology.

Duplicate service naming

Services that exist on both Atlantis and Calypso use prefixes:

  • [ATL] Sonarr, [ATL] Radarr, etc. for Atlantis
  • [CAL] Sonarr, [CAL] Radarr, etc. for Calypso

AdGuard DNS fix for *.tail.vish.gg on pi-5

Pi-5's Docker daemon was using 100.100.100.100 (Tailscale MagicDNS) but AdGuard on Calypso was forwarding *.vish.gg to Cloudflare, which returned stale IPs. Fixed by adding a private upstream in AdGuard config at /volume1/docker/adguard/config/AdGuardHome.yaml:

upstream_dns:
  - "[/tail.vish.gg/]100.100.100.100"

NPM Proxy Host Gotcha — Same-Subnet LAN IPs

Problem: NPM on Calypso (192.168.0.250) cannot reach Docker-published ports on other hosts that are on the same LAN subnet (192.168.0.x).

Root cause: When the Tailscale_outbound_connections DSM task runs tailscale configure-host on Calypso, it installs kernel netfilter hooks. After this, Docker containers on Calypso sending traffic to a LAN IP on the same subnet bypass the DNAT rules on the destination host (same-subnet traffic doesn't go through PREROUTING on the target). The containers are unreachable via their published ports.

Fix: Always use the Tailscale IP as the forward_host in NPM for services running in Docker on other hosts, not the LAN IP.

Host Use this in NPM (not LAN IP)
Homelab VM 100.67.40.126
Guava / TrueNAS 100.75.252.64
Atlantis 100.83.230.112

Why it worked pre-Headscale: Before the migration, Tailscale on Calypso ran in pure userspace mode without kernel netfilter hooks. NPM's outbound packets took the normal kernel path, hitting the destination's Docker DNAT rules correctly. The configure-host task (which installs kernel hooks) is required for Headscale's subnet routing to work, which introduced this side effect.

Known affected proxy hosts (already fixed to Tailscale IPs):

  • gf.vish.gg → 100.67.40.126:3300 (Grafana)
  • ntfy.vish.gg → 100.67.40.126:8081 (NTFY)
  • hoarder.thevish.io → 100.67.40.126:3482 (Karakeep)
  • binterest.thevish.io → 100.67.40.126:21544 (Binternet)
  • crista.love → 100.75.252.64:28888 (Guava nginx/static site)

DERP Relay Servers

Three DERP relay regions are configured for redundancy:

Region Code Host DERP Port STUN Port Notes
900 home-cal headscale.vish.gg:8443 8443 none Headscale built-in, LAN only
901 sea derp-sea.vish.gg:8444 8444 3478 Seattle VPS
902 home-atl derp-atl.vish.gg:8445 8445 3480 Atlantis NAS — added for redundancy

Important

: Tailscale public DERP servers (sfo, nyc, etc.) are disabled. Headscale nodes cannot authenticate through Tailscale's infrastructure. All relay traffic goes through regions 900, 901, or 902.

DERP Infrastructure Notes

  • derp-sea.vish.gg → Seattle VPS (YOUR_WAN_IP), derper container at hosts/vms/seattle/derper.yaml
  • derp-atl.vish.gg → Home public IP (184.23.52.14), router forwards 8445/tcp + 3480/udp to Atlantis (192.168.0.200)
    • Container deployed as Portainer stack ID 688 on Atlantis (from hosts/synology/atlantis/derper.yaml)
    • TLS cert at /volume1/docker/derper-atl/certs/live/derp-atl.vish.gg/ (flat .crt/.key layout required by derper)
    • Cloudflare credentials at /volume1/docker/derper-atl/secrets/cloudflare.ini
    • Cert auto-renewed monthly (1st of month, 03:00) by derper-atl-cert-renewer sidecar container (certbot/dns-cloudflare + supercronic; logs at /volume1/docker/derper-atl/certs/renew.log)
  • Port 3478/udp: coturn/Jitsi on Atlantis — do not use
  • Port 3479/udp: coturn/Matrix TURN on matrix-ubuntu — do not use
  • derpmap.yaml lives at hosts/synology/calypso/derpmap.yaml in repo; must be manually synced to /volume1/docker/headscale/config/derpmap.yaml on Calypso after changes

Subnet Routes in Use

Subnet Advertised by Approved
192.168.0.0/24 calypso (primary), atlantis ✅
192.168.68.0/22 vish-concord-nuc ✅
192.168.69.0/24 setillo ✅
192.168.12.0/24 gl-mt3000 ✅

Node Inventory

ID Hostname Tailscale IP Status
1 headscale-test 100.64.0.1 test LXC
2 seattle (vmi2076105) 100.82.197.124 Seattle VPS
3 matrix-ubuntu 100.85.21.51
4 pi-5 100.77.151.40
5 vish-concord-nuc 100.72.55.21
6 setillo 100.125.0.20
7 pve 100.87.12.28
8 truenas-scale 100.75.252.64 Guava/TrueNAS
9 ipad-pro 100.68.71.48
10 iphone16-pro-max 100.79.252.108
11 atlantis 100.83.230.112
12 calypso 100.103.48.78 Runs headscale
13 homelab 100.67.40.126
14 uqiyoe 100.124.91.52 Windows laptop
15 jellyfish 100.69.121.120 Remote location
16 gl-mt3000 100.126.243.15 Remote router
17 gl-be3600 100.105.59.123 Home router

Still to migrate (offline nodes)

Run tailscale up --login-server=https://headscale.vish.gg:8443 --force-reauth when they come online:

  • kevinlaptop (100.89.160.65)
  • mah-pc (100.121.22.51)
  • shinku-ryuu (100.98.93.15)
  • vish-mint (100.115.169.43)
  • vishdebian (100.86.60.62)
  • mastodon-rocky (100.111.200.21)
  • nvidia-shield (100.89.79.99)
  • pi-5-kevin (100.123.246.75)
  • rocky9-playground (100.105.250.128)
  • samsung-sm-x510 (100.72.118.117)
  • sd (100.83.141.1)
  • bluecrownpassionflower (100.110.25.127)
  • glkvm (100.64.137.1)
  • google-pixel-10-pro (100.122.119.40)

Home Assistant — Migrated ✅

Device: Home Assistant Green at 192.168.12.202:8123 (jellyfish remote location) Tailscale IP: 100.112.186.90 (preserved) | Node ID: 19 | MagicDNS: homeassistant.tail.vish.gg

Migration completed remotely (no physical access needed) via:

  1. HA WebSocket API (ws://192.168.12.202:8123/api/websocket) proxied through jellyfish (100.69.121.120)
  2. Supervisor addon_configs API to set login_server: https://headscale.vish.gg:8443
  3. Uninstalled + reinstalled the Tailscale add-on to clear stale tailscaled.state (necessary because can't change --login-server without --force-reauth)
  4. Add-on registered against headscale — auth URL approved via headscale nodes register
  5. IP updated via SQLite: UPDATE nodes SET ipv4='100.112.186.90' WHERE id=19;

Current add-on config:

{ "login_server": "https://headscale.vish.gg:8443", "accept_dns": false }

Uptime Kuma monitor: [JLF] Home Assistant (ID 5) → homeassistant.tail.vish.gg:8123

HA API token (expires 2028-06-07): eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiIxMzA1ZTE0NDg2ZGY0NDExYmMyOGEwZTY3ZmUyMTc3NyIsImlhdCI6MTc3MzA1MjkzNywiZXhwIjoyMDg4NDEyOTM3fQ.hzqjg7ALTdTDkMJS9Us-RUetQ309Nmfzx4gXevRRlp8

Outstanding TODOs

Priority Task Notes
Low Migrate offline nodes ~13 nodes still on Tailscale cloud — migrate when they come online
Info NPM proxy hosts audit Going forward, always use Tailscale IPs in NPM for Docker services on other LAN hosts (see NPM section above)
Reference in New Issue View Git Blame Copy Permalink