homelab-optimized/docs/troubleshooting/matrix-ssl-authentik-incident-2026-03-19.md

Matrix SSL + Authentik Proxy Incident — 2026-03-19

---

Issues Addressed

1. mx.vish.gg "Not Secure" Warning

Symptom: Browser showed "Not Secure" on https://mx.vish.gg.

Root cause: NPM was serving the Cloudflare Origin Certificate (cert ID 1, *.vish.gg) for mx.vish.gg. Cloudflare Origin certs are only trusted by Cloudflare's edge — since mx.vish.gg is unproxied (required for Matrix federation), browsers hit the origin directly and don't trust the cert.

Fix:

1. Got a proper Let's Encrypt cert for mx.vish.gg via Cloudflare DNS challenge on matrix-ubuntu:

   sudo certbot certonly --dns-cloudflare \
     --dns-cloudflare-credentials /etc/cloudflare.ini \
     -d mx.vish.gg --email your-email@example.com --agree-tos
   

2. Copied cert to NPM as npm-6:

   /volume1/docker/nginx-proxy-manager/data/custom_ssl/npm-6/fullchain.pem
   /volume1/docker/nginx-proxy-manager/data/custom_ssl/npm-6/privkey.pem
   

3. Updated NPM proxy host 10 (mx.vish.gg) to use cert ID 6
4. Set up renewal hook: /etc/letsencrypt/renewal-hooks/deploy/copy-to-npm.sh

Same fix applied for: livekit.mx.vish.gg (cert npm-7, proxy host 47)

---

2. kuma.vish.gg Redirect Loop (ERR_TOO_MANY_REDIRECTS)

Symptom: kuma.vish.gg (Uptime Kuma) caused infinite redirect loop via Authentik Forward Auth.

Root cause (two issues):

Issue A — Missing X-Original-URL header: The Authentik outpost returned 500 for Forward Auth requests because NPM wasn't passing the X-Original-URL header. The outpost log showed:

failed to detect a forward URL from nginx

Fix: Added to NPM advanced config for kuma.vish.gg (proxy host 41):

auth_request /outpost.goauthentik.io/auth/nginx;
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;

Issue B — Empty cookie_domain on all Forward Auth providers: After login, Authentik couldn't set the session cookie correctly because cookie_domain was empty on all proxy providers. This caused the auth loop to continue even after successful authentication.

Fix: Set cookie_domain: vish.gg on all proxy providers via Authentik API:

PK	Provider	Was	Now
4	Paperless Forward Auth	''	vish.gg
5	vish.gg Domain Forward Auth	vish.gg	✅ already set
8	Scrutiny Forward Auth	''	vish.gg
12	Uptime Kuma Forward Auth	''	vish.gg
13	Ollama Forward Auth	''	vish.gg
14	Wizarr Forward Auth	''	vish.gg

AK_TOKEN="..."
for pk in 4 8 12 13 14; do
  PROVIDER=$(curl -s "https://sso.vish.gg/api/v3/providers/proxy/$pk/" -H "Authorization: Bearer $AK_TOKEN")
  UPDATED=$(echo "$PROVIDER" | python3 -c "import sys,json; d=json.load(sys.stdin); d['cookie_domain']='vish.gg'; print(json.dumps(d))")
  curl -s -X PUT "https://sso.vish.gg/api/v3/providers/proxy/$pk/" \
    -H "Authorization: Bearer $AK_TOKEN" -H "Content-Type: application/json" -d "$UPDATED"
done

---

3. TURN Server External Verification

coturn was verified working externally from Seattle VPS (different network):

Test	Result
UDP port 3479 reachable	✅
STUN Binding request	✅ 0x0101 success, returns 184.23.52.14:3479
TURN Allocate (auth required)	✅ 0x0113 (401) — server responds, relay functional

Config: /etc/turnserver.conf on matrix-ubuntu

• listening-port=3479
• use-auth-secret
• static-auth-secret = same as turn_shared_secret in Synapse homeserver.yaml
• realm=matrix.thevish.io

---

NPM Certificate Reference

Cert ID	Nice Name	Domain	Type	Expires	Notes
1	Cloudflare Origin - vish.gg	*.vish.gg, vish.gg	Cloudflare Origin	2041	Only trusted by CF edge — don't use for unproxied
2	Cloudflare Origin - thevish.io	*.thevish.io	Cloudflare Origin	2026	Same caveat
3	Cloudflare Origin - crista.love	*.crista.love	Cloudflare Origin	2026	Same caveat
4	git.vish.gg (LE)	git.vish.gg	Let's Encrypt	2026-05
5	headscale.vish.gg (LE)	headscale.vish.gg	Let's Encrypt	2026-06
6	mx.vish.gg (LE)	mx.vish.gg	Let's Encrypt	2026-06	Added 2026-03-19
7	livekit.mx.vish.gg (LE)	livekit.mx.vish.gg	Let's Encrypt	2026-06	Added 2026-03-19

Rule: Any domain that is unproxied in Cloudflare (DNS-only, orange cloud off) must use a real Let's Encrypt cert, not the Cloudflare Origin cert.

---

Renewal Automation

Certs 6 and 7 are issued by certbot on matrix-ubuntu and auto-renewed via systemd timer. Deploy hooks copy renewed certs to NPM on Calypso:

/etc/letsencrypt/renewal-hooks/deploy/copy-to-npm.sh

To manually renew and deploy:

ssh matrix-ubuntu
sudo certbot renew --force-renewal -d mx.vish.gg
# hook runs automatically and copies to NPM

Last updated: 2026-03-19