homelab-optimized/docs/runbooks/credential-rotation.md

Credential Rotation Runbook

Overview

Step-by-step rotation procedures for all credentials exposed in the homelab-optimized public mirror (audited 2026-02-20). Work through each section in priority order. After updating secrets in compose files, commit and push — GitOps will redeploy automatically.

Note: Almost all of these stem from the same root cause — secrets were hard-coded in compose files, then those files were committed to git, then generate_service_docs.py and wiki-upload scripts duplicated those secrets into documentation, creating 3–5× copies of every secret across the repo. See the "Going Forward" section for how to prevent this.

Prerequisites

• SSH / Tailscale access to Atlantis, Calypso, Homelab VM, Seattle VM, matrix-ubuntu-vm
• Gitea admin access (git.vish.gg)
• Authentik admin access
• Google account access (Gmail app passwords)
• Cloudflare dashboard access
• OpenAI platform access
• Write access to this repository

Metadata

• Estimated Time: 4–6 hours
• Risk Level: Medium (service restarts required for most items)
• Requires Downtime: Brief per-service restart only
• Reversible: Yes (old values can be restored if something breaks)
• Last Updated: 2026-02-20

---

Priority 1 — Rotate Immediately (Externally Usable Tokens)

1. Gitea API Tokens

Two tokens hard-coded across scripts and docs.

1a. Wiki/scripts token (77e3ddaf...)

Files to update:

• scripts/cleanup-gitea-wiki.sh
• scripts/upload-all-docs-to-gitea-wiki.sh
• scripts/upload-to-gitea-wiki.sh
• scripts/create-clean-organized-wiki.sh
• scripts/upload-organized-wiki.sh
• docs/admin/DOCUMENTATION_MAINTENANCE_GUIDE.md

# 1. Go to https://git.vish.gg/user/settings/applications
# 2. Revoke the token starting 77e3ddaf
# 3. Generate new token, name: homelab-wiki, scope: repo
# 4. Replace in all files:
NEW_TOKEN=REDACTED_TOKEN
for f in scripts/cleanup-gitea-wiki.sh \
          scripts/upload-all-docs-to-gitea-wiki.sh \
          scripts/upload-to-gitea-wiki.sh \
          scripts/create-clean-organized-wiki.sh \
          scripts/upload-organized-wiki.sh \
          docs/admin/DOCUMENTATION_MAINTENANCE_GUIDE.md; do
  sed -i "s/REDACTED_GITEA_TOKEN/$NEW_TOKEN/g" "$f"
done

1b. Retro-site clone token (52fa6ccb...)

File: Calypso/retro-site.yaml and hosts/synology/calypso/retro-site.yaml

# 1. Go to https://git.vish.gg/user/settings/applications
# 2. Revoke the token starting 52fa6ccb
# 3. Generate new token, name: retro-site-deploy, scope: repo:read
# 4. Update the git clone URL in both compose files
# Consider switching to a deploy key for least-privilege access

---

2. Cloudflare API Token (FGXlHM7doB8Z...)

Appears in 13 files including active dynamic DNS updaters on multiple hosts.

Files to update (active deployments):

• hosts/synology/atlantis/dynamicdnsupdater.yaml
• hosts/physical/guava/portainer_yaml/dynamic_dns.yaml
• hosts/physical/concord-nuc/dyndns_updater.yaml
• Various Calypso/homelab-vm DDNS configs

Files to sanitize (docs):

• docs/infrastructure/cloudflare-dns.md
• docs/infrastructure/npm-migration-jan2026.md
• Any docs/services/individual/ddns-*.md files

# 1. Go to https://dash.cloudflare.com/profile/api-tokens
# 2. Find the token (FGXlHM7doB8Z...) and click Revoke
# 3. Create a new token: use "Edit zone DNS" template, scope to your zone only
# 4. Replace in all compose files above
# 5. Replace hardcoded value in docs with: YOUR_CLOUDFLARE_API_TOKEN

# Verify DDNS containers restart and can still update DNS:
docker logs cloudflare-ddns --tail 20

---

3. OpenAI API Key (sk-proj-C_IYp6io...)

Files to update:

• hosts/vms/homelab-vm/hoarder.yaml
• docs/services/individual/web.md (replace with placeholder)

# 1. Go to https://platform.openai.com/api-keys
# 2. Delete the exposed key
# 3. Create a new key, set a usage limit
# 4. Update OPENAI_API_KEY in hoarder.yaml
# 5. Replace value in docs with: YOUR_OPENAI_API_KEY

---

Priority 2 — OAuth / SSO Secrets

4. Grafana ↔ Authentik OAuth Secret

Files to update:

• hosts/vms/homelab-vm/monitoring.yaml
• hosts/synology/atlantis/grafana.yml
• docs/infrastructure/authentik-sso.md (replace with placeholder)
• docs/services/individual/grafana-oauth.md (replace with placeholder)

# 1. Log into Authentik admin: https://auth.vish.gg/if/admin/
# 2. Applications → Providers → find Grafana OAuth2 provider
# 3. Edit → regenerate Client Secret → copy both Client ID and Secret
# 4. Update in both compose files:
#    GF_AUTH_GENERIC_OAUTH_CLIENT_ID: NEW_ID
#    GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: NEW_SECRET
# 5. Commit and push — both Grafana stacks restart automatically

# Verify SSO works after restart:
curl -I https://gf.vish.gg

---

5. Seafile ↔ Authentik OAuth Secret

Files to update:

• hosts/synology/calypso/seafile-oauth-config.py
• docs/services/individual/seafile-oauth.md (replace with placeholder)

# 1. Log into Authentik admin
# 2. Applications → Providers → find Seafile OAuth2 provider
# 3. Regenerate client secret
# 4. Update OAUTH_CLIENT_ID and OAUTH_CLIENT_SECRET in seafile-oauth-config.py
# 5. Re-run the config script on the Seafile server to apply

---

6. Authentik Secret Key (RpRexcYo5HAz...)

Critical — this key encrypts all Authentik data (tokens, sessions, stored credentials).

File: hosts/synology/calypso/authentik/docker-compose.yaml

# 1. Generate a new secret:
python3 -c "import secrets; print(secrets.token_urlsafe(50))"

# 2. Update AUTHENTIK_SECRET_KEY in docker-compose.yaml
# 3. Commit and push — Authentik will restart
# WARNING: All active Authentik sessions will be invalidated.
#          Users will need to log back in. SSO-protected services
#          may temporarily show login errors while Authentik restarts.

# Verify Authentik is healthy after restart:
docker logs authentik_server --tail 30

---

Priority 3 — Application Secrets (Require Service Restart)

7. Gmail App Passwords

Five distinct app passwords were found across the repo. Revoke all of them in Google Account → Security → App passwords, then create new per-service ones.

Password	Used For	Active Files
(see Vaultwarden)	Mastodon, Joplin, Authentik SMTP	matrix-ubuntu-vm/mastodon/.env.production.template, atlantis/joplin.yml, calypso/authentik/docker-compose.yaml
(see Vaultwarden)	Vaultwarden SMTP	atlantis/vaultwarden.yaml
(see Vaultwarden)	Documenso SMTP	atlantis/documenso/documenso.yaml
(see Vaultwarden)	Reactive Resume v4 (archived)	archive/reactive_resume_v4_archived/docker-compose.yml
(see Vaultwarden)	Reactive Resume v5 (active)	calypso/reactive_resume_v5/docker-compose.yml

Best practice: Create one app password per service, named clearly (e.g., homelab-joplin, homelab-mastodon). Update each file's SMTP_PASS / SMTP_PASSWORD / MAILER_AUTH_PASSWORD / smtp_password field.

---

8. Matrix Synapse Secrets

Three secrets in homeserver.yaml, plus the TURN shared secret.

File: hosts/synology/atlantis/matrix_synapse_docs/homeserver.yaml

# Generate fresh values for each:
python3 -c "import secrets; print(secrets.token_urlsafe(48))"

# Fields to rotate:
#   registration_shared_secret
#   macaroon_secret_key
#   form_secret
#   turn_shared_secret

# After updating homeserver.yaml, restart Synapse:
docker restart synapse   # or via Portainer

# Also update coturn config on the server directly:
ssh atlantis
nano /path/to/turnserver.conf
# Update: static-auth-secret=NEW_TURN_SECRET
systemctl restart coturn

# Update instructions.txt — replace old values with REDACTED

---

9. Mastodon SECRET_KEY_BASE + OTP_SECRET

File: hosts/synology/atlantis/mastodon.yml Also in: docs/services/individual/mastodon.md (replace with placeholder)

# Generate new values:
openssl rand -hex 64   # for SECRET_KEY_BASE
openssl rand -hex 64   # for OTP_SECRET

# Update both in mastodon.yml
# Commit and push — GitOps restarts Mastodon
# WARNING: All active user sessions are invalidated. Users must log back in.

# Verify Mastodon web is accessible:
curl -I https://your-mastodon-domain/
docker logs mastodon_web --tail 20

---

10. Documenso Secrets (3 keys)

Files:

• hosts/synology/atlantis/documenso/documenso.yaml
• hosts/synology/atlantis/documenso/Secrets.txt (will be removed by sanitizer)
• docs/services/individual/documenso.md (replace with placeholder)

# Generate new values:
python3 -c "import secrets; print(secrets.token_urlsafe(32))"  # NEXTAUTH_SECRET
python3 -c "import secrets; print(secrets.token_urlsafe(32))"  # NEXT_PRIVATE_ENCRYPTION_KEY
python3 -c "import secrets; print(secrets.token_urlsafe(32))"  # NEXT_PRIVATE_ENCRYPTION_SECONDARY_KEY

# Update all three in documenso.yaml
# NOTE: Rotating encryption keys will invalidate signed documents.
#       Confirm this is acceptable before rotating.

---

11. Paperless-NGX API Token

Files:

• hosts/synology/calypso/paperless/paperless-ai.yml
• hosts/synology/calypso/paperless/README.md (replace with placeholder)
• docs/services/paperless.md (replace with placeholder)

# 1. Log into Paperless web UI
# 2. Admin → Auth Token → delete existing, generate new
# 3. Update PAPERLESS_API_TOKEN in paperless-ai.yml
# 4. Commit and push

---

12. Immich JWT Secret (Both NAS)

Files:

• hosts/synology/atlantis/immich/stack.env (will be removed by sanitizer)
• hosts/synology/calypso/immich/stack.env (will be removed by sanitizer)

Since these files are removed by the sanitizer, ensure they are in .gitignore or managed via Portainer env variables going forward.

# Generate new secret:
openssl rand -base64 96

# Update JWT_SECRET in both stack.env files locally,
# then apply via Portainer (not committed to git).
# WARNING: All active Immich sessions invalidated.

---

13. Revolt/Stoatchat — LiveKit API Secret + VAPID Private Key

Files:

• hosts/vms/seattle/stoatchat/livekit.yml
• hosts/vms/seattle/stoatchat/Revolt.overrides.toml
• hosts/vms/homelab-vm/stoatchat.yaml
• docs/services/stoatchat/Revolt.overrides.toml (replace with placeholder)
• hosts/vms/seattle/stoatchat/DEPLOYMENT_SUMMARY.md (replace with placeholder)

# Generate new LiveKit API key/secret pair:
# Use the LiveKit CLI or generate random strings:
python3 -c "import secrets; print(secrets.token_urlsafe(24))"  # API key
python3 -c "import secrets; print(secrets.token_urlsafe(32))"  # API secret

# Generate new VAPID key pair:
npx web-push generate-vapid-keys
# or: python3 -c "from py_vapid import Vapid; v=Vapid(); v.generate_keys(); print(v.private_key)"

# Update in livekit.yml and Revolt.overrides.toml
# Restart LiveKit and Revolt services

---

14. Jitsi Internal Auth Passwords (6 passwords)

File: hosts/synology/atlantis/jitsi/jitsi.yml Also in: hosts/synology/atlantis/jitsi/.env (will be removed by sanitizer)

# Generate new passwords for each variable:
for var in JICOFO_COMPONENT_SECRET JICOFO_AUTH_PASSWORD JVB_AUTH_PASSWORD \
           JIGASI_XMPP_PASSWORD JIBRI_RECORDER_PASSWORD JIBRI_XMPP_PASSWORD; do
  echo "$var=$(openssl rand -hex 10)"
done

# Update all 6 in jitsi.yml
# Restart the entire Jitsi stack — all components must use the same passwords
docker compose -f jitsi.yml down && docker compose -f jitsi.yml up -d

---

15. SNMP v3 Auth + Priv Passwords

Used for NAS monitoring — same credentials across 6 files.

Files to update:

• hosts/synology/setillo/prometheus/snmp.yml
• hosts/synology/atlantis/grafana_prometheus/snmp.yml
• hosts/synology/atlantis/grafana_prometheus/snmp_mariushosting.yml
• hosts/synology/calypso/grafana_prometheus/snmp.yml
• hosts/vms/homelab-vm/monitoring.yaml

# 1. Log into each Synology NAS DSM
# 2. Go to Control Panel → Terminal & SNMP → SNMP tab
# 3. Update SNMPv3 auth password and privacy password to new values
# 4. Update the same values in all 5 config files above
# 5. The archive file (deprecated-monitoring-stacks) can just be left for
#    the sanitizer to redact.

---

16. Invidious hmac_key

Files:

• hosts/physical/concord-nuc/invidious/invidious.yaml
• hosts/physical/concord-nuc/invidious/invidious_old/invidious.yaml
• hosts/synology/atlantis/invidious.yml

# Generate new hmac_key:
python3 -c "import secrets; print(secrets.token_hex(16))"

# Update hmac_key in each active invidious.yaml
# Restart Invidious containers

---

17. Open WebUI Secret Keys

Files:

• hosts/vms/contabo-vm/ollama/docker-compose.yml
• hosts/synology/atlantis/ollama/docker-compose.yml
• hosts/synology/atlantis/ollama/64_bit_key.txt (will be removed by sanitizer)

# Generate new key:
openssl rand -hex 32

# Update WEBUI_SECRET_KEY in both compose files
# Restart Open WebUI containers — active sessions invalidated

---

18. Portainer Edge Key

File: hosts/vms/homelab-vm/portainer_agent.yaml

# 1. Log into Portainer at https://192.168.0.200:9443
# 2. Go to Settings → Edge Compute → Edge Agents
# 3. Find the homelab-vm agent and regenerate its edge key
# 4. Update EDGE_KEY in portainer_agent.yaml with the new base64 value
# 5. Restart the Portainer edge agent container

---

19. OpenProject Secret Key

File: hosts/vms/homelab-vm/openproject.yml Also in: docs/services/individual/openproject.md (replace with placeholder)

openssl rand -hex 64
# Update OPENPROJECT_SECRET_KEY_BASE in openproject.yml
# Restart OpenProject — sessions invalidated

---

20. RomM Auth Secret Key

File: hosts/vms/homelab-vm/romm/romm.yaml Also: hosts/vms/homelab-vm/romm/secret_key.yaml (will be removed by sanitizer)

openssl rand -hex 32
# Update ROMM_AUTH_SECRET_KEY in romm.yaml
# Restart RomM — sessions invalidated

---

21. Hoarder NEXTAUTH Secret

File: hosts/vms/homelab-vm/hoarder.yaml Also in: docs/services/individual/web.md (replace with placeholder)

openssl rand -base64 36
# Update NEXTAUTH_SECRET in hoarder.yaml
# Restart Hoarder — sessions invalidated

---

Priority 4 — Shared / Weak Passwords

22. REDACTED_PASSWORD123! — Used Across 5+ Services

This password is the same for all of the following. Change each to a unique strong password:

Service	File	Variable
NetBox	hosts/synology/atlantis/netbox.yml	SUPERUSER_PASSWORD
Paperless admin	hosts/synology/calypso/paperless/docker-compose.yml	PAPERLESS_ADMIN_PASSWORD
Seafile admin	hosts/synology/calypso/seafile-server.yaml	INIT_SEAFILE_ADMIN_PASSWORD
Seafile admin (new)	hosts/synology/calypso/seafile-new.yaml	INIT_SEAFILE_ADMIN_PASSWORD
PhotoPrism	hosts/physical/anubis/photoprism.yml	PHOTOPRISM_ADMIN_PASSWORD
Hemmelig	hosts/vms/bulgaria-vm/hemmelig.yml	SECRET_JWT_SECRET
Vaultwarden admin	hosts/synology/atlantis/bitwarden/bitwarden_token.txt	(source password)

For each: generate openssl rand -base64 18, update in the compose file, restart the container, then log in to verify.

---

23. REDACTED_PASSWORD — Used Across 3 Services

Service	File	Variable
Gotify	hosts/vms/homelab-vm/gotify.yml	GOTIFY_DEFAULTUSER_PASS
Pi-hole	hosts/synology/atlantis/pihole.yml	WEBPASSWORD
Stirling PDF	hosts/synology/atlantis/stirlingpdf.yml	SECURITY_INITIAL_LOGIN_PASSWORD

---

24. mastodon_pass_2026 — Live PostgreSQL Password

Files:

• hosts/vms/matrix-ubuntu-vm/mastodon/.env.production.template
• hosts/vms/matrix-ubuntu-vm/docs/SETUP.md

# On the matrix-ubuntu-vm server:
ssh YOUR_WAN_IP
sudo -u postgres psql
ALTER USER mastodon WITH PASSWORD 'REDACTED_PASSWORD';
\q

# Update the password in .env.production.template and Mastodon's running config
# Restart Mastodon services

---

25. Watchtower API Token (REDACTED_WATCHTOWER_TOKEN)

File
hosts/synology/atlantis/watchtower.yml
hosts/synology/calypso/prometheus.yml

# Generate a proper random token:
openssl rand -hex 20
# Update WATCHTOWER_HTTP_API_TOKEN in both files
# Update any scripts that call the Watchtower API

---

26. test:test SSH Credentials on YOUR_WAN_IP

The matrix-ubuntu-vm CREDENTIALS.md shows a test user with password test.

# SSH to the server and remove or secure the test account:
ssh YOUR_WAN_IP
passwd test          # change to a strong password
# or: userdel -r test   # remove entirely if unused

---

Priority 5 — Network Infrastructure

27. Management Switch Password Hashes

File: mgmtswitch.conf (will be removed from public mirror by sanitizer)

The SHA-512 hashes for root, vish, and vkhemraj switch accounts are crackable offline. Rotate the switch passwords:

# SSH to the management switch
ssh admin@10.0.0.15
# Change passwords for all local accounts:
enable
configure terminal
username root secret NEW_PASSWORD
username vish secret NEW_PASSWORD
username vkhemraj secret NEW_PASSWORD
write memory

---

Final Verification

After completing all rotations:

# 1. Commit and push all file changes
git add -A
git commit -m "chore(security): rotate all exposed credentials"
git push origin main

# 2. Wait for the mirror workflow to complete, then pull:
git -C /home/homelab/organized/repos/homelab-optimized pull

# 3. Verify none of the old secrets appear in the public mirror:
cd /home/homelab/organized/repos/homelab-optimized
grep -r "77e3ddaf\|52fa6ccb\|FGXlHM7d\|sk-proj-C_IYp6io\|ArP5XWdkwVyw\|bdtrpmpce\|toiunzuby" . 2>/dev/null
grep -r "244c619d\|RpRexcYo5\|mastodon_pass\|REDACTED_PASSWORD\|REDACTED_PASSWORD\|REDACTED_WATCHTOWER_TOKEN" . 2>/dev/null
grep -r "2e80b1b7d3a\|eca299ae59\|rxmr4tJoqfu\|ZjCofRlfm6\|QE5SudhZ99" . 2>/dev/null
# All should return no results

# 4. Verify GitOps deployments are healthy in Portainer:
#    https://192.168.0.200:9443

---

Going Forward — Preventing This Again

The root cause: secrets hard-coded in compose files that get committed to git.

Rules:

1. Never hard-code secrets in compose files — use Docker Secrets, or an .env file excluded by .gitignore (Portainer can load env files from the host at deploy time)
2. Never put real values in documentation — use YOUR_API_KEY placeholders
3. Never create Secrets.txt or CREDENTIALS.md files in the repo — use a password manager (you already have Vaultwarden/Bitwarden)
4. Run the sanitizer locally before any commit that touches secrets:

# Test in a temp copy — see what the sanitizer would catch:
tmpdir=$(mktemp -d)
cp -r /path/to/homelab "$tmpdir/"
python3 "$tmpdir/homelab/.gitea/sanitize.py"

Related Documentation

• Security Hardening
• Repository Sanitization
• GitOps Deployment Guide

Portainer Git Credential Rotation

The saved Git credential portainer-homelab (credId: 1) is used by ~43 stacks to pull compose files from git.vish.gg. When the Gitea token expires or is rotated, all those stacks fail to redeploy.

# 1. Generate a new Gitea token at https://git.vish.gg/user/settings/applications
#    Scope: read:repository

# 2. Test the token:
curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: token YOUR_NEW_TOKEN" \
  "https://git.vish.gg/api/v1/repos/Vish/homelab"
# Should return 200

# 3. Update in Portainer:
curl -k -s -X PUT \
  -H "X-API-Key: "REDACTED_API_KEY" \
  -H "Content-Type: application/json" \
  "https://192.168.0.200:9443/api/users/1/gitcredentials/1" \
  -d '{"name":"portainer-homelab","username":"vish","password":"YOUR_NEW_TOKEN"}'

Note: The API update may not immediately propagate to automated pulls. Pass credentials inline in redeploy calls to force use of the new token.

---

Change Log

• 2026-02-27 — Incident: sanitization commit 037d766a replaced credentials with REDACTED_PASSWORD placeholders across 14 compose files. All affected containers detected via Portainer API env scan and restored from git show 037d766a^. Added Portainer Git credential rotation section above.
• 2026-02-20 — Initial creation (8 items)
• 2026-02-20 — Expanded after full private repo audit (27 items across 34 exposure categories)