Sanitized mirror from private repository - 2026-03-29 13:33:25 UTC

2026-03-29 13:33:25 +00:00
commit 75d4f4e02b
1280 changed files with 331190 additions and 0 deletions
--- a/docs/infrastructure/offline-and-remote-access.md
+++ b/docs/infrastructure/offline-and-remote-access.md
@@ -0,0 +1,271 @@
+# Offline & Remote Access Guide
+
+Last updated: 2026-03-20
+
+## How DNS Resolution Works
+
+The homelab uses **split-horizon DNS** so services are reachable from anywhere — LAN, Tailscale VPN, or the open internet — using the same `*.vish.gg` domain names.
+
+### Three Access Paths
+
+```
+┌──────────────────────────────────────────────────────────────────────┐
+│                     DNS Query: nb.vish.gg                            │
+├──────────────┬──────────────────┬────────────────────────────────────┤
+│  LAN Client  │  Tailscale Client│  Internet Client                   │
+│  (at home)   │  (travel laptop) │  (phone on cellular)               │
+├──────────────┼──────────────────┼────────────────────────────────────┤
+│  DNS: AdGuard│  DNS: Headscale  │  DNS: Cloudflare                   │
+│  (192.168.0  │  MagicDNS →      │  (1.1.1.1)                         │
+│   .250)      │  AdGuard         │                                    │
+├──────────────┼──────────────────┼────────────────────────────────────┤
+│  Resolves to:│  Resolves to:    │  Resolves to:                      │
+│  100.85.21.51│  100.85.21.51    │  104.21.73.214 (Cloudflare)        │
+│  (NPM via TS)│  (NPM via TS)   │                                    │
+├──────────────┼──────────────────┼────────────────────────────────────┤
+│  Path:       │  Path:           │  Path:                             │
+│  Client →    │  Client →        │  Client → Cloudflare →             │
+│  NPM (direct)│  Tailscale →     │  Router → NPM →                   │
+│  → backend   │  NPM → backend   │  backend                          │
+├──────────────┼──────────────────┼────────────────────────────────────┤
+│  Latency:    │  Latency:        │  Latency:                          │
+│  ~1ms        │  ~5-50ms         │  ~50-100ms                         │
+│  (LAN)       │  (Tailscale)     │  (Cloudflare roundtrip)            │
+├──────────────┼──────────────────┼────────────────────────────────────┤
+│  Internet    │  Internet        │  Internet                          │
+│  required?   │  required?       │  required?                         │
+│  NO          │  NO (peer-to-peer│  YES                               │
+│              │  if both on TS)  │                                    │
+└──────────────┴──────────────────┴────────────────────────────────────┘
+```
+
+### Key: Everything Resolves to 100.85.21.51
+
+All `*.vish.gg`, `*.thevish.io`, and `*.crista.love` domains resolve to `100.85.21.51` (matrix-ubuntu's Tailscale IP) when queried through AdGuard. This is NPM's address on the Tailscale network, reachable from:
+
+- **LAN clients** — via the router's DHCP DNS (AdGuard at 192.168.0.250)
+- **Remote Tailscale clients** — via Headscale MagicDNS which forwards to AdGuard
+- **Both paths hit NPM on its Tailscale IP**, which works from anywhere on the tailnet
+
+## When Internet Goes Down
+
+If your WAN link drops:
+
+| What works | How |
+|------------|-----|
+| All `*.vish.gg` services | AdGuard returns Tailscale IP, NPM proxies locally |
+| MagicDNS names (`atlantis.tail.vish.gg`) | Headscale resolves directly |
+| Direct Tailscale IPs (100.x.x.x) | Always work between peers |
+| Olares/K8s (k9s, kubectl) | LAN access at 192.168.0.145 |
+
+| What breaks | Why |
+|-------------|-----|
+| External access (from internet) | Cloudflare can't reach you |
+| Cloudflare-only domains without split-horizon rewrite | DNS returns unreachable CF proxy IP |
+| Renovate, DDNS updates | Need internet to reach APIs |
+| DERP relays for remote peers | Remote Tailscale clients may lose connectivity |
+
+## Access from Travel Laptop
+
+Your travel laptop (MSI Prestige) connects via Headscale VPN:
+
+1. **Join the tailnet**: `tailscale up --login-server=https://headscale.vish.gg`
+2. **DNS is automatic**: Headscale pushes AdGuard as the DNS server via MagicDNS
+3. **All domains work**: `nb.vish.gg`, `git.vish.gg`, etc. resolve to NPM's Tailscale IP
+4. **No VPN split tunneling needed**: Only homelab traffic routes through Tailscale
+
+```bash
+# From the travel laptop:
+curl https://nb.vish.gg/       # → 100.85.21.51 (Tailscale) → NPM → backend
+curl https://gf.vish.gg/       # → 100.85.21.51 (Tailscale) → NPM → Grafana
+ssh homelab.tail.vish.gg       # → MagicDNS → direct Tailscale peer
+```
+
+### If Headscale Is Down
+
+If the Headscale control server (calypso) is unreachable, already-connected peers maintain their connections. New peers can't join. Use direct Tailscale IPs as fallback:
+
+| Service | Direct URL |
+|---------|-----------|
+| Grafana | `http://100.67.40.126:3300` |
+| NetBox | `http://100.67.40.126:8443` |
+| Portainer | `https://100.83.230.112:9443` |
+| Gitea | `http://100.103.48.78:3052` |
+
+## MagicDNS (.tail.vish.gg)
+
+Headscale MagicDNS provides `<hostname>.tail.vish.gg` for all peers:
+
+| Hostname | Tailscale IP | Use |
+|----------|-------------|-----|
+| atlantis.tail.vish.gg | 100.83.230.112 | NAS, media |
+| calypso.tail.vish.gg | 100.103.48.78 | NAS, Gitea, auth |
+| homelab.tail.vish.gg | 100.67.40.126 | Monitoring, tools |
+| matrix-ubuntu.tail.vish.gg | 100.85.21.51 | NPM, Matrix, Mastodon |
+| pve.tail.vish.gg | 100.87.12.28 | Proxmox |
+| pi-5.tail.vish.gg | 100.77.151.40 | Uptime Kuma |
+| vish-concord-nuc.tail.vish.gg | 100.72.55.21 | Home Assistant, edge |
+| setillo.tail.vish.gg | 100.125.0.20 | Remote NAS |
+| seattle.tail.vish.gg | 100.82.197.124 | Cloud VPS |
+| truenas-scale.tail.vish.gg | 100.75.252.64 | TrueNAS |
+
+`.tail.vish.gg` names are resolved by AdGuard rewrites (not MagicDNS) so they work on **all LAN devices**, not just Tailscale clients. Both AdGuard instances (Calypso and Atlantis) have identical entries.
+
+### .vish.local Names
+
+AdGuard also resolves `.vish.local` shortnames to Tailscale IPs:
+
+| Hostname | Tailscale IP |
+|----------|-------------|
+| atlantis.vish.local | 100.83.230.112 |
+| calypso.vish.local | 100.103.48.78 |
+| homelab.vish.local | 100.67.40.126 |
+| concordnuc.vish.local | 100.72.55.21 |
+| pi5.vish.local | 100.77.151.40 |
+| px.vish.local | 100.87.12.28 |
+
+## DNS Infrastructure
+
+### Two Redundant AdGuard Instances
+
+Both instances have **identical configuration** — same rewrites, filters, upstream DNS, and user rules.
+
+| Role | Host | IP | Web UI |
+|------|------|-----|--------|
+| **Primary DNS** | Calypso | `192.168.0.250` | `http://192.168.0.250:9080` |
+| **Backup DNS** | Atlantis | `192.168.0.200` | `http://192.168.0.200:9080` |
+
+Router DHCP hands out both as DNS servers. If Calypso reboots, Atlantis takes over seamlessly.
+
+Login for both: username `vish`, same password.
+
+### Upstream DNS
+
+Both AdGuard instances use:
+- `https://dns.adguard-dns.com/dns-query` (AdGuard DoH)
+- `https://dns.cloudflare.com/dns-query` (Cloudflare DoH)
+- `[/tail.vish.gg/]100.100.100.100` (Headscale MagicDNS for tail.vish.gg)
+
+### AdGuard DNS Rewrites (Split-Horizon)
+
+All rewrites are identical on both Calypso and Atlantis.
+
+**Wildcard rewrites (all services through NPM):**
+
+| Domain Pattern | Resolves To | Purpose |
+|---------------|-------------|---------|
+| `*.vish.gg` | `100.85.21.51` | NPM via Tailscale |
+| `*.thevish.io` | `100.85.21.51` | NPM via Tailscale |
+| `*.crista.love` | `100.85.21.51` | NPM via Tailscale |
+
+**Specific overrides (bypass NPM wildcard):**
+
+| Domain | Resolves To | Purpose |
+|--------|-------------|---------|
+| `derp.vish.gg` | `192.168.0.250` | DERP relay — direct, no NPM |
+| `derp-atl.vish.gg` | `192.168.0.200` | DERP relay — direct, no NPM |
+| `derp-sea.vish.gg` | `100.82.197.124` | DERP relay on Seattle VPS |
+| `turn.thevish.io` | `192.168.0.200` | TURN/STUN — needs direct UDP |
+
+**Tailscale host rewrites (override *.vish.gg wildcard):**
+
+| Domain | Resolves To |
+|--------|-------------|
+| `atlantis.tail.vish.gg` | `100.83.230.112` |
+| `calypso.tail.vish.gg` | `100.103.48.78` |
+| `homelab.tail.vish.gg` | `100.67.40.126` |
+| `matrix-ubuntu.tail.vish.gg` | `100.85.21.51` |
+| `pve.tail.vish.gg` | `100.87.12.28` |
+| `pi-5.tail.vish.gg` | `100.77.151.40` |
+| `vish-concord-nuc.tail.vish.gg` | `100.72.55.21` |
+| `setillo.tail.vish.gg` | `100.125.0.20` |
+| `seattle.tail.vish.gg` | `100.82.197.124` |
+| `truenas-scale.tail.vish.gg` | `100.75.252.64` |
+| `jellyfish.tail.vish.gg` | `100.69.121.120` |
+| `shinku-ryuu.tail.vish.gg` | `100.98.93.15` |
+
+### Keeping Both Instances in Sync
+
+When adding new DNS rewrites, update **both** AdGuard configs:
+- Calypso: `/volume1/docker/adguard/config/AdGuardHome.yaml`
+- Atlantis: `/volume1/docker/adguard/config/AdGuardHome.yaml`
+
+Then restart both:
+```bash
+ssh calypso "sudo docker restart AdGuard"
+ssh atlantis "sudo /var/packages/REDACTED_APP_PASSWORD/target/usr/bin/docker restart AdGuard"
+```
+
+### Ad-Blocking Filters
+
+Both instances use the same 5 filter lists:
+1. AdGuard DNS filter
+2. AdAway Default Blocklist
+3. AdGuard DNS Popup Hosts filter
+4. Dandelion Sprout's Anti Push Notifications
+5. AWAvenue Ads Rule
+
+Plus 20 custom user rules blocking specific ad domains.
+
+## SSL Certificates
+
+All services use **Let's Encrypt wildcard certificates** (issued via DNS challenge with Cloudflare API):
+
+| Certificate | Domains | Issuer |
+|------------|---------|--------|
+| Cert 8 | `*.vish.gg`, `vish.gg` | ZeroSSL (via acme.sh) |
+| Cert 9 | `*.thevish.io`, `thevish.io` | ZeroSSL (via acme.sh) |
+| Cert 10 | `*.crista.love`, `crista.love` | ZeroSSL (via acme.sh) |
+
+These certs are **publicly trusted** — no certificate warnings on any access path (LAN, Tailscale, or internet).
+
+### Certificate Renewal
+
+acme.sh is installed on matrix-ubuntu (`/home/test/.acme.sh/`) with auto-renewal via cron. To manually renew:
+
+```bash
+ssh matrix-ubuntu
+export CF_Token="REDACTED_TOKEN"  # pragma: allowlist secret
+~/.acme.sh/acme.sh --renew -d '*.vish.gg' -d 'vish.gg' --force
+~/.acme.sh/acme.sh --renew -d '*.thevish.io' -d 'thevish.io' --force
+~/.acme.sh/acme.sh --renew -d '*.crista.love' -d 'crista.love' --force
+
+# Then re-upload to NPM (certs need to be uploaded via NPM API or UI)
+```
+
+## Quick Reference
+
+### I'm at home on WiFi
+Just use `https://nb.vish.gg` — AdGuard resolves to NPM's Tailscale IP, works instantly.
+
+### I'm traveling with the laptop
+Connect to Headscale tailnet → same URLs work: `https://nb.vish.gg`
+
+### I'm on my phone (no VPN)
+Use the public URLs: `https://nb.vish.gg` → goes through Cloudflare as normal.
+
+### Internet is down at home
+All services still work from LAN via AdGuard → Tailscale IP → NPM. No Cloudflare dependency.
+
+### I need to access a service directly (no NPM)
+Three options, all equivalent:
+```
+http://homelab.tail.vish.gg:3300      # .tail.vish.gg name
+http://homelab.vish.local:3300        # .vish.local shortname
+http://100.67.40.126:3300             # Tailscale IP directly
+```
+
+### Everything is down — emergency access
+SSH via Tailscale: `ssh homelab` (uses ~/.ssh/config with Tailscale IPs)
+
+### I need to manage DNS
+- Calypso AdGuard: `http://192.168.0.250:9080` (primary)
+- Atlantis AdGuard: `http://192.168.0.200:9080` (backup)
+- Login: `vish` / same password on both
+
+## Related Documentation
+
+- [Split-Horizon DNS Implementation](split-horizon-dns.md)
+- [NPM Migration Plan](npm-migration-to-matrix-ubuntu.md)
+- [Authentik SSO](authentik-sso.md)
+- [Image Update Guide](../admin/IMAGE_UPDATE_GUIDE.md)