# Headscale Migration Guide

## Overview

This homelab uses a self-hosted [Headscale](https://github.com/juanfont/headscale) instance instead of Tailscale cloud. Headscale is a drop-in open-source replacement for the Tailscale control server.

- **Headscale server**: `https://headscale.vish.gg:8443`
- **MagicDNS suffix**: `tail.vish.gg` (e.g. `atlantis.tail.vish.gg`)
- **Login**: Authentik SSO at `sso.vish.gg` — username `vish` or email `admin@thevish.io`
- **Hosted on**: Calypso (`192.168.0.250`), managed via Docker

---

## Connecting a New Device

### Linux (Ubuntu / Debian)

1. Install Tailscale if not already installed:
   ```bash
   curl -fsSL https://tailscale.com/install.sh | sh
   ```

2. Connect to headscale:
   ```bash
   sudo tailscale up \
     --login-server=https://headscale.vish.gg:8443 \
     --accept-routes \
     --force-reauth
   ```

3. A browser auth URL will be printed. Open it and log in with Authentik SSO.

4. If DNS doesn't resolve `headscale.vish.gg` (e.g. fresh machine with no AdGuard), add a temporary hosts entry first:
   ```bash
   echo '184.23.52.14 headscale.vish.gg' | sudo tee -a /etc/hosts
   # Run tailscale up, then clean up:
   sudo sed -i '/headscale.vish.gg/d' /etc/hosts
   ```

5. If the machine was previously on Tailscale cloud and complains about non-default flags, Tailscale will print the exact command with all required flags — copy and run that command.

> **Note**: After registration, an admin must approve the node and fix the IP if preserving the original Tailscale IP (see Admin section below).

---

### Windows

1. Download and install Tailscale from https://tailscale.com/download/windows

2. Open **PowerShell as Administrator** and run:
   ```powershell
   tailscale up --login-server=https://headscale.vish.gg:8443 --accept-routes --force-reauth
   ```

3. A browser window will open — log in with Authentik SSO (`vish` / `admin@thevish.io`).

4. If it shows a "mention all non-default flags" error, copy and run the exact command it provides, adding `--login-server=https://headscale.vish.gg:8443 --force-reauth` to it.

> **Important**: Always include `--accept-routes` on Windows otherwise subnet routes (e.g. `192.168.0.x`) won't be reachable.

---

### iOS (iPhone / iPad)

1. Install **Tailscale** from the App Store.

2. Open the app → tap your **account icon** (top right) → **Log in**

3. Tap the `···` menu (top right of the login screen) → **Use custom coordination server**

4. Enter: `https://headscale.vish.gg:8443` → **Save**

5. Log in with Authentik SSO — username `vish` or email `admin@thevish.io`

> **Note**: `.vish.local` hostnames do NOT work on iOS — iOS intercepts `.local` for mDNS and never forwards to DNS. Use Tailscale IPs (`100.x.x.x`) or MagicDNS names (`hostname.tail.vish.gg`) instead.

---

### macOS

1. Install Tailscale from the App Store or https://tailscale.com/download/mac

2. **Option A — GUI**: Click the Tailscale menu bar icon → Preferences → hold `Option` while clicking "Log in" to enter a custom server URL → enter `https://headscale.vish.gg:8443`

3. **Option B — CLI**:
   ```bash
   sudo tailscale up \
     --login-server=https://headscale.vish.gg:8443 \
     --accept-routes \
     --force-reauth
   ```

4. Log in with Authentik SSO when the browser opens.

> **Note**: Same as iOS, `.vish.local` hostnames won't resolve on macOS when remote. Use `hostname.tail.vish.gg` or the Tailscale IP instead.

---

### GL.iNet Routers (OpenWrt)

1. SSH into the router.

2. Add a hosts entry (since GL routers don't use AdGuard):
   ```bash
   echo '184.23.52.14 headscale.vish.gg' >> /etc/hosts
   ```

3. Run tailscale up — it will error with the required flags. Copy and run the exact command it provides, appending:
   ```
   --login-server=https://headscale.vish.gg:8443 --auth-key=<preauth-key> --force-reauth
   ```
   Get a pre-auth key from an admin (see below).

4. If advertising subnet routes, add `--advertise-routes=<subnet>` to the command.

---

### Home Assistant (Tailscale Add-on)

> **Note**: HA Green does not expose SSH by default. Use the WebSocket API approach below,
> which works fully remotely via a Tailscale-connected hop host.

**Remote migration steps** (no physical access required):

1. Reach HA via a hop host on the same LAN (e.g. jellyfish at `100.69.121.120`):
   ```
   ssh lulu@100.69.121.120
   curl http://192.168.12.202:8123/api/  # confirm HA reachable
   ```

2. If the add-on was previously authenticated to Tailscale cloud, it will refuse
   `--login-server` change with: `can't change --login-server without --force-reauth`.
   **Fix**: uninstall + reinstall the add-on via supervisor API to clear `tailscaled.state`:
   ```python
   # Via HA WebSocket API (supervisor/api endpoint):
   {"type": "supervisor/api", "endpoint": "/addons/a0d7b954_tailscale/uninstall", "method": "post"}
   {"type": "supervisor/api", "endpoint": "/addons/a0d7b954_tailscale/install", "method": "post"}
   ```

3. Set options before starting:
   ```python
   {"type": "supervisor/api", "endpoint": "/addons/a0d7b954_tailscale/options", "method": "post",
    "data": {"options": {"login_server": "https://headscale.vish.gg:8443", "accept_dns": false}}}
   ```

4. Start the add-on via `hassio/addon_start` service, then read logs:
   ```
   GET http://192.168.12.202:8123/api/hassio/addons/a0d7b954_tailscale/logs
   ```
   Look for: `AuthURL is https://headscale.vish.gg:8443/register/<key>`

5. Register on Calypso:
   ```bash
   docker exec headscale headscale nodes register --user vish --key <key-from-log>
   ```

6. Fix IP via SQLite (see section above) and restart headscale.

---

## Admin: Registering a New Node

After a node connects, an admin needs to:

### 1. Generate a Pre-Auth Key (optional, avoids browser auth)

```bash
ssh -p 62000 Vish@192.168.0.250
sudo /volume1/@appstore/REDACTED_APP_PASSWORD/usr/bin/docker exec headscale \
  headscale preauthkeys create --user 1 --expiration 1h
```

Use `--authkey=<key>` instead of browser auth in `tailscale up`.

### 2. Check Registered Nodes

```bash
sudo /volume1/@appstore/REDACTED_APP_PASSWORD/usr/bin/docker exec headscale headscale nodes list
```

### 3. Preserve Original Tailscale IP (if migrating from Tailscale cloud)

Headscale v0.28+ removed the `--ipv4` flag. Fix IPs via SQLite:

```bash
sudo sqlite3 /volume1/@docker/volumes/headscale-data/_data/db.sqlite \
  "UPDATE nodes SET ipv4='100.x.x.x' WHERE id=<node-id>;"
sudo /volume1/@appstore/REDACTED_APP_PASSWORD/usr/bin/docker restart headscale
```

### 4. Rename a Node

```bash
sudo /volume1/@appstore/REDACTED_APP_PASSWORD/usr/bin/docker exec headscale \
  headscale nodes rename -i <id> <new-name>
```

### 5. Approve Subnet Routes

Routes advertised by nodes must be explicitly approved:

```bash
sudo /volume1/@appstore/REDACTED_APP_PASSWORD/usr/bin/docker exec headscale \
  headscale nodes approve-routes -i <node-id> -r <subnet>
# e.g. -r 192.168.0.0/24
```

Check all routes (v0.28 — routes are embedded in node JSON output):
```bash
sudo /volume1/@appstore/REDACTED_APP_PASSWORD/usr/bin/docker exec headscale \
  headscale nodes list --output json | python3 -c "
import sys,json
for n in json.load(sys.stdin):
    r=n.get('available_routes',[])
    a=n.get('approved_routes',[])
    if r: print(n['given_name'], 'available:', r, 'approved:', a)
"
```

---

## DNS Notes

- **MagicDNS**: Headscale pushes `192.168.0.250` (Calypso AdGuard) as DNS to all tailnet clients
- **AdGuard rewrites**: `*.vish.local` names resolve to their Tailscale IPs via AdGuard rewrites on Calypso
- **`.vish.local` on iOS/macOS**: Does NOT work remotely — iOS/macOS intercept `.local` for mDNS. Use `hostname.tail.vish.gg` instead
- **External DNS**: `headscale.vish.gg` resolves to `184.23.52.14` (home WAN) externally, `192.168.0.250` internally via AdGuard rewrite

## Uptime Kuma Monitoring

Kuma runs on **pi-5** (`100.77.151.40`) inside the `uptime-kuma` container. DB at `/app/data/kuma.db`.

### Monitor groups and hosts

| Group | Host | Tailscale IP |
|-------|------|-------------|
| Homelab | `homelab.tail.vish.gg` | `100.67.40.126` |
| Atlantis | `atlantis.tail.vish.gg` | `100.83.230.112` |
| Calypso | `calypso.tail.vish.gg` | `100.103.48.78` |
| Concord_NUC | `vish-concord-nuc.tail.vish.gg` | `100.72.55.21` |
| Setillo | `setillo.tail.vish.gg` | `100.125.0.20` |
| Proxmox_NUC | `pve.tail.vish.gg` | `100.87.12.28` |
| Guava | `truenas-scale.tail.vish.gg` | `100.75.252.64` |
| Seattle | `seattle.tail.vish.gg` | `100.82.197.124` |
| Raspberry Pi 5 | `100.77.151.40` | `100.77.151.40` |

### Firewall rules required for Kuma (pi-5 = `100.77.151.40`)

Kuma polls via Tailscale IP. Each host with a ts-input/ts-forward chain needs ACCEPT rules for pi-5:

- **Homelab VM**: Rules in `iptables-legacy` ts-input/ts-forward for pi-5 on all monitored ports. Persisted via `netfilter-persistent`.
- **Concord NUC**: Same — ts-input/ts-forward ACCEPT for pi-5 on monitored ports.
- **Seattle**: UFW rule `ufw allow from 100.77.151.40 to any port 8444`
- **Calypso/Atlantis/Setillo**: No ts-input blocking — Tailscale is in userspace mode on Synology.

### Duplicate service naming

Services that exist on both Atlantis and Calypso use prefixes:
- `[ATL] Sonarr`, `[ATL] Radarr`, etc. for Atlantis
- `[CAL] Sonarr`, `[CAL] Radarr`, etc. for Calypso

### AdGuard DNS fix for `*.tail.vish.gg` on pi-5

Pi-5's Docker daemon was using `100.100.100.100` (Tailscale MagicDNS) but AdGuard on Calypso was forwarding `*.vish.gg` to Cloudflare, which returned stale IPs. Fixed by adding a private upstream in AdGuard config at `/volume1/docker/adguard/config/AdGuardHome.yaml`:

```yaml
upstream_dns:
  - "[/tail.vish.gg/]100.100.100.100"
```

---

## NPM Proxy Host Gotcha — Same-Subnet LAN IPs

**Problem**: NPM on Calypso (`192.168.0.250`) cannot reach Docker-published ports on other hosts
that are on the same LAN subnet (`192.168.0.x`).

**Root cause**: When the `Tailscale_outbound_connections` DSM task runs `tailscale configure-host`
on Calypso, it installs kernel netfilter hooks. After this, Docker containers on Calypso sending
traffic to a LAN IP on the same subnet bypass the DNAT rules on the destination host (same-subnet
traffic doesn't go through PREROUTING on the target). The containers are unreachable via their
published ports.

**Fix**: Always use the **Tailscale IP** as the `forward_host` in NPM for services running in
Docker on other hosts, not the LAN IP.

| Host | Use this in NPM (not LAN IP) |
|------|------------------------------|
| Homelab VM | `100.67.40.126` |
| Guava / TrueNAS | `100.75.252.64` |
| Atlantis | `100.83.230.112` |

**Why it worked pre-Headscale**: Before the migration, Tailscale on Calypso ran in pure userspace
mode without kernel netfilter hooks. NPM's outbound packets took the normal kernel path, hitting
the destination's Docker DNAT rules correctly. The `configure-host` task (which installs kernel
hooks) is required for Headscale's subnet routing to work, which introduced this side effect.

**Known affected proxy hosts** (already fixed to Tailscale IPs):
- `gf.vish.gg` → `100.67.40.126:3300` (Grafana)
- `ntfy.vish.gg` → `100.67.40.126:8081` (NTFY)
- `hoarder.thevish.io` → `100.67.40.126:3482` (Karakeep)
- `binterest.thevish.io` → `100.67.40.126:21544` (Binternet)
- `crista.love` → `100.75.252.64:28888` (Guava nginx/static site)

---

## DERP Relay Servers

Three DERP relay regions are configured for redundancy:

| Region | Code | Host | DERP Port | STUN Port | Notes |
|--------|------|------|-----------|-----------|-------|
| 900 | home-cal | headscale.vish.gg:8443 | 8443 | none | Headscale built-in, LAN only |
| 901 | sea | derp-sea.vish.gg:8444 | 8444 | 3478 | Seattle VPS |
| 902 | home-atl | derp-atl.vish.gg:8445 | 8445 | 3480 | Atlantis NAS — added for redundancy |

> **Important**: Tailscale public DERP servers (sfo, nyc, etc.) are disabled. Headscale nodes cannot authenticate through Tailscale's infrastructure. All relay traffic goes through regions 900, 901, or 902.

### DERP Infrastructure Notes

- `derp-sea.vish.gg` → Seattle VPS (`YOUR_WAN_IP`), derper container at `hosts/vms/seattle/derper.yaml`
- `derp-atl.vish.gg` → Home public IP (`184.23.52.14`), router forwards `8445/tcp` + `3480/udp` to Atlantis (`192.168.0.200`)
  - Container deployed as **Portainer stack ID 688** on Atlantis (from `hosts/synology/atlantis/derper.yaml`)
  - TLS cert at `/volume1/docker/derper-atl/certs/live/derp-atl.vish.gg/` (flat `.crt`/`.key` layout required by derper)
  - Cloudflare credentials at `/volume1/docker/derper-atl/secrets/cloudflare.ini`
  - Cert auto-renewed monthly (1st of month, 03:00) by `derper-atl-cert-renewer` sidecar container
    (certbot/dns-cloudflare + supercronic; logs at `/volume1/docker/derper-atl/certs/renew.log`)
- Port 3478/udp: coturn/Jitsi on Atlantis — do not use
- Port 3479/udp: coturn/Matrix TURN on matrix-ubuntu — do not use
- `derpmap.yaml` lives at `hosts/synology/calypso/derpmap.yaml` in repo; must be manually synced to `/volume1/docker/headscale/config/derpmap.yaml` on Calypso after changes

## Subnet Routes in Use

| Subnet | Advertised by | Approved |
|--------|--------------|---------|
| 192.168.0.0/24 | calypso (primary), atlantis | ✅ |
| 192.168.68.0/22 | vish-concord-nuc | ✅ |
| 192.168.69.0/24 | setillo | ✅ |
| 192.168.12.0/24 | gl-mt3000 | ✅ |

## Node Inventory

| ID | Hostname | Tailscale IP | Status |
|----|----------|-------------|--------|
| 1 | headscale-test | 100.64.0.1 | test LXC |
| 2 | seattle (vmi2076105) | 100.82.197.124 | Seattle VPS |
| 3 | matrix-ubuntu | 100.85.21.51 | |
| 4 | pi-5 | 100.77.151.40 | |
| 5 | vish-concord-nuc | 100.72.55.21 | |
| 6 | setillo | 100.125.0.20 | |
| 7 | pve | 100.87.12.28 | |
| 8 | truenas-scale | 100.75.252.64 | Guava/TrueNAS |
| 9 | ipad-pro | 100.68.71.48 | |
| 10 | iphone16-pro-max | 100.79.252.108 | |
| 11 | atlantis | 100.83.230.112 | |
| 12 | calypso | 100.103.48.78 | Runs headscale |
| 13 | homelab | 100.67.40.126 | |
| 14 | uqiyoe | 100.124.91.52 | Windows laptop |
| 15 | jellyfish | 100.69.121.120 | Remote location |
| 16 | gl-mt3000 | 100.126.243.15 | Remote router |
| 17 | gl-be3600 | 100.105.59.123 | Home router |

### Still to migrate (offline nodes)
Run `tailscale up --login-server=https://headscale.vish.gg:8443 --force-reauth` when they come online:

- kevinlaptop (`100.89.160.65`)
- mah-pc (`100.121.22.51`)
- shinku-ryuu (`100.98.93.15`)
- vish-mint (`100.115.169.43`)
- vishdebian (`100.86.60.62`)
- mastodon-rocky (`100.111.200.21`)
- nvidia-shield (`100.89.79.99`)
- pi-5-kevin (`100.123.246.75`)
- rocky9-playground (`100.105.250.128`)
- samsung-sm-x510 (`100.72.118.117`)
- sd (`100.83.141.1`)
- bluecrownpassionflower (`100.110.25.127`)
- glkvm (`100.64.137.1`)
- google-pixel-10-pro (`100.122.119.40`)

### Home Assistant — Migrated ✅

**Device**: Home Assistant Green at `192.168.12.202:8123` (jellyfish remote location)
**Tailscale IP**: `100.112.186.90` (preserved) | **Node ID**: 19 | **MagicDNS**: `homeassistant.tail.vish.gg`

**Migration completed** remotely (no physical access needed) via:
1. HA WebSocket API (`ws://192.168.12.202:8123/api/websocket`) proxied through jellyfish (`100.69.121.120`)
2. Supervisor `addon_configs` API to set `login_server: https://headscale.vish.gg:8443`
3. Uninstalled + reinstalled the Tailscale add-on to clear stale `tailscaled.state`
   (necessary because `can't change --login-server without --force-reauth`)
4. Add-on registered against headscale — auth URL approved via `headscale nodes register`
5. IP updated via SQLite: `UPDATE nodes SET ipv4='100.112.186.90' WHERE id=19;`

**Current add-on config**:
```json
{ "login_server": "https://headscale.vish.gg:8443", "accept_dns": false }
```

**Uptime Kuma monitor**: `[JLF] Home Assistant` (ID 5) → `homeassistant.tail.vish.gg:8123`

**HA API token** (expires 2028-06-07):
`eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiIxMzA1ZTE0NDg2ZGY0NDExYmMyOGEwZTY3ZmUyMTc3NyIsImlhdCI6MTc3MzA1MjkzNywiZXhwIjoyMDg4NDEyOTM3fQ.hzqjg7ALTdTDkMJS9Us-RUetQ309Nmfzx4gXevRRlp8`  <!-- pragma: allowlist secret -->

---

## Outstanding TODOs

| Priority | Task | Notes |
|----------|------|-------|
| Low | **Migrate offline nodes** | ~13 nodes still on Tailscale cloud — migrate when they come online |
| Info | **NPM proxy hosts audit** | Going forward, always use Tailscale IPs in NPM for Docker services on other LAN hosts (see NPM section above) |
