homelab-optimized/docs/runbooks/credential-rotation.md

# Credential Rotation Runbook

## Overview

Step-by-step rotation procedures for all credentials exposed in the
`homelab-optimized` public mirror (audited 2026-02-20). Work through each
section in priority order. After updating secrets in compose files, commit
and push — GitOps will redeploy automatically.

> **Note:** Almost all of these stem from the same root cause — secrets were
> hard-coded in compose files, then those files were committed to git, then
> `generate_service_docs.py` and wiki-upload scripts duplicated those secrets
> into documentation, creating 3–5× copies of every secret across the repo.
> See the "Going Forward" section for how to prevent this.

## Prerequisites

- [ ] SSH / Tailscale access to Atlantis, Calypso, Homelab VM, Seattle VM, matrix-ubuntu-vm
- [ ] Gitea admin access (`git.vish.gg`)
- [ ] Authentik admin access
- [ ] Google account access (Gmail app passwords)
- [ ] Cloudflare dashboard access
- [ ] OpenAI platform access
- [ ] Write access to this repository

## Metadata

- **Estimated Time**: 4–6 hours
- **Risk Level**: Medium (service restarts required for most items)
- **Requires Downtime**: Brief per-service restart only
- **Reversible**: Yes (old values can be restored if something breaks)
- **Last Updated**: 2026-02-20

---

## Priority 1 — Rotate Immediately (Externally Usable Tokens)

### 1. Gitea API Tokens

Two tokens hard-coded across scripts and docs.

#### 1a. Wiki/scripts token (`77e3ddaf...`)

**Files to update:**
- `scripts/cleanup-gitea-wiki.sh`
- `scripts/upload-all-docs-to-gitea-wiki.sh`
- `scripts/upload-to-gitea-wiki.sh`
- `scripts/create-clean-organized-wiki.sh`
- `scripts/upload-organized-wiki.sh`
- `docs/admin/DOCUMENTATION_MAINTENANCE_GUIDE.md`

```bash
# 1. Go to https://git.vish.gg/user/settings/applications
# 2. Revoke the token starting 77e3ddaf
# 3. Generate new token, name: homelab-wiki, scope: repo
# 4. Replace in all files:
NEW_TOKEN=REDACTED_TOKEN
for f in scripts/cleanup-gitea-wiki.sh \
          scripts/upload-all-docs-to-gitea-wiki.sh \
          scripts/upload-to-gitea-wiki.sh \
          scripts/create-clean-organized-wiki.sh \
          scripts/upload-organized-wiki.sh \
          docs/admin/DOCUMENTATION_MAINTENANCE_GUIDE.md; do
  sed -i "s/REDACTED_GITEA_TOKEN/$NEW_TOKEN/g" "$f"
done
```

#### 1b. Retro-site clone token (`52fa6ccb...`)

**File:** `Calypso/retro-site.yaml` and `hosts/synology/calypso/retro-site.yaml`

```bash
# 1. Go to https://git.vish.gg/user/settings/applications
# 2. Revoke the token starting 52fa6ccb
# 3. Generate new token, name: retro-site-deploy, scope: repo:read
# 4. Update the git clone URL in both compose files
# Consider switching to a deploy key for least-privilege access
```

---

### 2. Cloudflare API Token (`FGXlHM7doB8Z...`)

Appears in 13 files including active dynamic DNS updaters on multiple hosts.

**Files to update (active deployments):**
- `hosts/synology/atlantis/dynamicdnsupdater.yaml`
- `hosts/physical/guava/portainer_yaml/dynamic_dns.yaml`
- `hosts/physical/concord-nuc/dyndns_updater.yaml`
- Various Calypso/homelab-vm DDNS configs

**Files to sanitize (docs):**
- `docs/infrastructure/cloudflare-dns.md`
- `docs/infrastructure/npm-migration-jan2026.md`
- Any `docs/services/individual/ddns-*.md` files

```bash
# 1. Go to https://dash.cloudflare.com/profile/api-tokens
# 2. Find the token (FGXlHM7doB8Z...) and click Revoke
# 3. Create a new token: use "Edit zone DNS" template, scope to your zone only
# 4. Replace in all compose files above
# 5. Replace hardcoded value in docs with: YOUR_CLOUDFLARE_API_TOKEN

# Verify DDNS containers restart and can still update DNS:
docker logs cloudflare-ddns --tail 20
```

---

### 3. OpenAI API Key (`sk-proj-C_IYp6io...`)

**Files to update:**
- `hosts/vms/homelab-vm/hoarder.yaml`
- `docs/services/individual/web.md` (replace with placeholder)

```bash
# 1. Go to https://platform.openai.com/api-keys
# 2. Delete the exposed key
# 3. Create a new key, set a usage limit
# 4. Update OPENAI_API_KEY in hoarder.yaml
# 5. Replace value in docs with: YOUR_OPENAI_API_KEY
```

---

## Priority 2 — OAuth / SSO Secrets

### 4. Grafana ↔ Authentik OAuth Secret

**Files to update:**
- `hosts/vms/homelab-vm/monitoring.yaml`
- `hosts/synology/atlantis/grafana.yml`
- `docs/infrastructure/authentik-sso.md` (replace with placeholder)
- `docs/services/individual/grafana-oauth.md` (replace with placeholder)

```bash
# 1. Log into Authentik admin: https://auth.vish.gg/if/admin/
# 2. Applications → Providers → find Grafana OAuth2 provider
# 3. Edit → regenerate Client Secret → copy both Client ID and Secret
# 4. Update in both compose files:
#    GF_AUTH_GENERIC_OAUTH_CLIENT_ID: NEW_ID
#    GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: NEW_SECRET
# 5. Commit and push — both Grafana stacks restart automatically

# Verify SSO works after restart:
curl -I https://gf.vish.gg
```

---

### 5. Seafile ↔ Authentik OAuth Secret

**Files to update:**
- `hosts/synology/calypso/seafile-oauth-config.py`
- `docs/services/individual/seafile-oauth.md` (replace with placeholder)

```bash
# 1. Log into Authentik admin
# 2. Applications → Providers → find Seafile OAuth2 provider
# 3. Regenerate client secret
# 4. Update OAUTH_CLIENT_ID and OAUTH_CLIENT_SECRET in seafile-oauth-config.py
# 5. Re-run the config script on the Seafile server to apply
```

---

### 6. Authentik Secret Key (`RpRexcYo5HAz...`)

**Critical** — this key encrypts all Authentik data (tokens, sessions, stored credentials).

**File:** `hosts/synology/calypso/authentik/docker-compose.yaml`

```bash
# 1. Generate a new secret:
python3 -c "import secrets; print(secrets.token_urlsafe(50))"

# 2. Update AUTHENTIK_SECRET_KEY in docker-compose.yaml
# 3. Commit and push — Authentik will restart
# WARNING: All active Authentik sessions will be invalidated.
#          Users will need to log back in. SSO-protected services
#          may temporarily show login errors while Authentik restarts.

# Verify Authentik is healthy after restart:
docker logs authentik_server --tail 30
```

---

## Priority 3 — Application Secrets (Require Service Restart)

### 7. Gmail App Passwords

Five distinct app passwords were found across the repo. Revoke all of them
in Google Account → Security → App passwords, then create new per-service ones.

| Password | Used For | Active Files |
|----------|----------|-------------|
| (see Vaultwarden) | Mastodon, Joplin, Authentik SMTP | `matrix-ubuntu-vm/mastodon/.env.production.template`, `atlantis/joplin.yml`, `calypso/authentik/docker-compose.yaml` |
| (see Vaultwarden) | Vaultwarden SMTP | `atlantis/vaultwarden.yaml` |
| (see Vaultwarden) | Documenso SMTP | `atlantis/documenso/documenso.yaml` |
| (see Vaultwarden) | Reactive Resume v4 (archived) | `archive/reactive_resume_v4_archived/docker-compose.yml` |
| (see Vaultwarden) | Reactive Resume v5 (active) | `calypso/reactive_resume_v5/docker-compose.yml` |

**Best practice:** Create one app password per service, named clearly (e.g.,
`homelab-joplin`, `homelab-mastodon`). Update each file's `SMTP_PASS` /
`SMTP_PASSWORD` / `MAILER_AUTH_PASSWORD` / `smtp_password` field.

---

### 8. Matrix Synapse Secrets

Three secrets in `homeserver.yaml`, plus the TURN shared secret.

**File:** `hosts/synology/atlantis/matrix_synapse_docs/homeserver.yaml`

```bash
# Generate fresh values for each:
python3 -c "import secrets; print(secrets.token_urlsafe(48))"

# Fields to rotate:
#   registration_shared_secret
#   macaroon_secret_key
#   form_secret
#   turn_shared_secret

# After updating homeserver.yaml, restart Synapse:
docker restart synapse   # or via Portainer

# Also update coturn config on the server directly:
ssh atlantis
nano /path/to/turnserver.conf
# Update: static-auth-secret=NEW_TURN_SECRET
systemctl restart coturn

# Update instructions.txt — replace old values with REDACTED
```

---

### 9. Mastodon `SECRET_KEY_BASE` + `OTP_SECRET`

**File:** `hosts/synology/atlantis/mastodon.yml`
**Also in:** `docs/services/individual/mastodon.md` (replace with placeholder)

```bash
# Generate new values:
openssl rand -hex 64   # for SECRET_KEY_BASE
openssl rand -hex 64   # for OTP_SECRET

# Update both in mastodon.yml
# Commit and push — GitOps restarts Mastodon
# WARNING: All active user sessions are invalidated. Users must log back in.

# Verify Mastodon web is accessible:
curl -I https://your-mastodon-domain/
docker logs mastodon_web --tail 20
```

---

### 10. Documenso Secrets (3 keys)

**Files:**
- `hosts/synology/atlantis/documenso/documenso.yaml`
- `hosts/synology/atlantis/documenso/Secrets.txt` (will be removed by sanitizer)
- `docs/services/individual/documenso.md` (replace with placeholder)

```bash
# Generate new values:
python3 -c "import secrets; print(secrets.token_urlsafe(32))"  # NEXTAUTH_SECRET
python3 -c "import secrets; print(secrets.token_urlsafe(32))"  # NEXT_PRIVATE_ENCRYPTION_KEY
python3 -c "import secrets; print(secrets.token_urlsafe(32))"  # NEXT_PRIVATE_ENCRYPTION_SECONDARY_KEY

# Update all three in documenso.yaml
# NOTE: Rotating encryption keys will invalidate signed documents.
#       Confirm this is acceptable before rotating.
```

---

### 11. Paperless-NGX API Token

**Files:**
- `hosts/synology/calypso/paperless/paperless-ai.yml`
- `hosts/synology/calypso/paperless/README.md` (replace with placeholder)
- `docs/services/paperless.md` (replace with placeholder)

```bash
# 1. Log into Paperless web UI
# 2. Admin → Auth Token → delete existing, generate new
# 3. Update PAPERLESS_API_TOKEN in paperless-ai.yml
# 4. Commit and push
```

---

### 12. Immich JWT Secret (Both NAS)

**Files:**
- `hosts/synology/atlantis/immich/stack.env` (will be removed by sanitizer)
- `hosts/synology/calypso/immich/stack.env` (will be removed by sanitizer)

Since these files are removed by the sanitizer, ensure they are in `.gitignore`
or managed via Portainer env variables going forward.

```bash
# Generate new secret:
openssl rand -base64 96

# Update JWT_SECRET in both stack.env files locally,
# then apply via Portainer (not committed to git).
# WARNING: All active Immich sessions invalidated.
```

---

### 13. Revolt/Stoatchat — LiveKit API Secret + VAPID Private Key

**Files:**
- `hosts/vms/seattle/stoatchat/livekit.yml`
- `hosts/vms/seattle/stoatchat/Revolt.overrides.toml`
- `hosts/vms/homelab-vm/stoatchat.yaml`
- `docs/services/stoatchat/Revolt.overrides.toml` (replace with placeholder)
- `hosts/vms/seattle/stoatchat/DEPLOYMENT_SUMMARY.md` (replace with placeholder)

```bash
# Generate new LiveKit API key/secret pair:
# Use the LiveKit CLI or generate random strings:
python3 -c "import secrets; print(secrets.token_urlsafe(24))"  # API key
python3 -c "import secrets; print(secrets.token_urlsafe(32))"  # API secret

# Generate new VAPID key pair:
npx web-push generate-vapid-keys
# or: python3 -c "from py_vapid import Vapid; v=Vapid(); v.generate_keys(); print(v.private_key)"

# Update in livekit.yml and Revolt.overrides.toml
# Restart LiveKit and Revolt services
```

---

### 14. Jitsi Internal Auth Passwords (6 passwords)

**File:** `hosts/synology/atlantis/jitsi/jitsi.yml`
**Also in:** `hosts/synology/atlantis/jitsi/.env` (will be removed by sanitizer)

```bash
# Generate new passwords for each variable:
for var in JICOFO_COMPONENT_SECRET JICOFO_AUTH_PASSWORD JVB_AUTH_PASSWORD \
           JIGASI_XMPP_PASSWORD JIBRI_RECORDER_PASSWORD JIBRI_XMPP_PASSWORD; do
  echo "$var=$(openssl rand -hex 10)"
done

# Update all 6 in jitsi.yml
# Restart the entire Jitsi stack — all components must use the same passwords
docker compose -f jitsi.yml down && docker compose -f jitsi.yml up -d
```

---

### 15. SNMP v3 Auth + Priv Passwords

Used for NAS monitoring — same credentials across 6 files.

**Files to update:**
- `hosts/synology/setillo/prometheus/snmp.yml`
- `hosts/synology/atlantis/grafana_prometheus/snmp.yml`
- `hosts/synology/atlantis/grafana_prometheus/snmp_mariushosting.yml`
- `hosts/synology/calypso/grafana_prometheus/snmp.yml`
- `hosts/vms/homelab-vm/monitoring.yaml`

```bash
# 1. Log into each Synology NAS DSM
# 2. Go to Control Panel → Terminal & SNMP → SNMP tab
# 3. Update SNMPv3 auth password and privacy password to new values
# 4. Update the same values in all 5 config files above
# 5. The archive file (deprecated-monitoring-stacks) can just be left for
#    the sanitizer to redact.
```

---

### 16. Invidious `hmac_key`

**Files:**
- `hosts/physical/concord-nuc/invidious/invidious.yaml`
- `hosts/physical/concord-nuc/invidious/invidious_old/invidious.yaml`
- `hosts/synology/atlantis/invidious.yml`

```bash
# Generate new hmac_key:
python3 -c "import secrets; print(secrets.token_hex(16))"

# Update hmac_key in each active invidious.yaml
# Restart Invidious containers
```

---

### 17. Open WebUI Secret Keys

**Files:**
- `hosts/vms/contabo-vm/ollama/docker-compose.yml`
- `hosts/synology/atlantis/ollama/docker-compose.yml`
- `hosts/synology/atlantis/ollama/64_bit_key.txt` (will be removed by sanitizer)

```bash
# Generate new key:
openssl rand -hex 32

# Update WEBUI_SECRET_KEY in both compose files
# Restart Open WebUI containers — active sessions invalidated
```

---

### 18. Portainer Edge Key

**File:** `hosts/vms/homelab-vm/portainer_agent.yaml`

```bash
# 1. Log into Portainer at https://192.168.0.200:9443
# 2. Go to Settings → Edge Compute → Edge Agents
# 3. Find the homelab-vm agent and regenerate its edge key
# 4. Update EDGE_KEY in portainer_agent.yaml with the new base64 value
# 5. Restart the Portainer edge agent container
```

---

### 19. OpenProject Secret Key

**File:** `hosts/vms/homelab-vm/openproject.yml`
**Also in:** `docs/services/individual/openproject.md` (replace with placeholder)

```bash
openssl rand -hex 64
# Update OPENPROJECT_SECRET_KEY_BASE in openproject.yml
# Restart OpenProject — sessions invalidated
```

---

### 20. RomM Auth Secret Key

**File:** `hosts/vms/homelab-vm/romm/romm.yaml`
**Also:** `hosts/vms/homelab-vm/romm/secret_key.yaml` (will be removed by sanitizer)

```bash
openssl rand -hex 32
# Update ROMM_AUTH_SECRET_KEY in romm.yaml
# Restart RomM — sessions invalidated
```

---

### 21. Hoarder NEXTAUTH Secret

**File:** `hosts/vms/homelab-vm/hoarder.yaml`
**Also in:** `docs/services/individual/web.md` (replace with placeholder)

```bash
openssl rand -base64 36
# Update NEXTAUTH_SECRET in hoarder.yaml
# Restart Hoarder — sessions invalidated
```

---

## Priority 4 — Shared / Weak Passwords

### 22. `REDACTED_PASSWORD123!` — Used Across 5+ Services

This password is the same for all of the following. Change each to a
**unique** strong password:

| Service | File | Variable |
|---------|------|----------|
| NetBox | `hosts/synology/atlantis/netbox.yml` | `SUPERUSER_PASSWORD` |
| Paperless admin | `hosts/synology/calypso/paperless/docker-compose.yml` | `PAPERLESS_ADMIN_PASSWORD` |
| Seafile admin | `hosts/synology/calypso/seafile-server.yaml` | `INIT_SEAFILE_ADMIN_PASSWORD` |
| Seafile admin (new) | `hosts/synology/calypso/seafile-new.yaml` | `INIT_SEAFILE_ADMIN_PASSWORD` |
| PhotoPrism | `hosts/physical/anubis/photoprism.yml` | `PHOTOPRISM_ADMIN_PASSWORD` |
| Hemmelig | `hosts/vms/bulgaria-vm/hemmelig.yml` | `SECRET_JWT_SECRET` |
| Vaultwarden admin | `hosts/synology/atlantis/bitwarden/bitwarden_token.txt` | (source password) |

For each: generate `openssl rand -base64 18`, update in the compose file,
restart the container, then log in to verify.

---

### 23. `REDACTED_PASSWORD` — Used Across 3 Services

| Service | File | Variable |
|---------|------|----------|
| Gotify | `hosts/vms/homelab-vm/gotify.yml` | `GOTIFY_DEFAULTUSER_PASS` |
| Pi-hole | `hosts/synology/atlantis/pihole.yml` | `WEBPASSWORD` |
| Stirling PDF | `hosts/synology/atlantis/stirlingpdf.yml` | `SECURITY_INITIAL_LOGIN_PASSWORD` |

---

### 24. `mastodon_pass_2026` — Live PostgreSQL Password

**Files:**
- `hosts/vms/matrix-ubuntu-vm/mastodon/.env.production.template`
- `hosts/vms/matrix-ubuntu-vm/docs/SETUP.md`

```bash
# On the matrix-ubuntu-vm server:
ssh YOUR_WAN_IP
sudo -u postgres psql
ALTER USER mastodon WITH PASSWORD 'REDACTED_PASSWORD';
\q

# Update the password in .env.production.template and Mastodon's running config
# Restart Mastodon services
```

---

### 25. Watchtower API Token (`REDACTED_WATCHTOWER_TOKEN`)

| File |
|------|
| `hosts/synology/atlantis/watchtower.yml` |
| `hosts/synology/calypso/prometheus.yml` |

```bash
# Generate a proper random token:
openssl rand -hex 20
# Update WATCHTOWER_HTTP_API_TOKEN in both files
# Update any scripts that call the Watchtower API
```

---

### 26. `test:test` SSH Credentials on `YOUR_WAN_IP`

The matrix-ubuntu-vm CREDENTIALS.md shows a `test` user with password `test`.

```bash
# SSH to the server and remove or secure the test account:
ssh YOUR_WAN_IP
passwd test          # change to a strong password
# or: userdel -r test   # remove entirely if unused
```

---

## Priority 5 — Network Infrastructure

### 27. Management Switch Password Hashes

**File:** `mgmtswitch.conf` (will be removed from public mirror by sanitizer)

The SHA-512 hashes for `root`, `vish`, and `vkhemraj` switch accounts are
crackable offline. Rotate the switch passwords:

```bash
# SSH to the management switch
ssh admin@10.0.0.15
# Change passwords for all local accounts:
enable
configure terminal
username root secret NEW_PASSWORD
username vish secret NEW_PASSWORD
username vkhemraj secret NEW_PASSWORD
write memory
```

---

## Final Verification

After completing all rotations:

```bash
# 1. Commit and push all file changes
git add -A
git commit -m "chore(security): rotate all exposed credentials"
git push origin main

# 2. Wait for the mirror workflow to complete, then pull:
git -C /home/homelab/organized/repos/homelab-optimized pull

# 3. Verify none of the old secrets appear in the public mirror:
cd /home/homelab/organized/repos/homelab-optimized
grep -r "77e3ddaf\|52fa6ccb\|FGXlHM7d\|sk-proj-C_IYp6io\|ArP5XWdkwVyw\|bdtrpmpce\|toiunzuby" . 2>/dev/null
grep -r "244c619d\|RpRexcYo5\|mastodon_pass\|REDACTED_PASSWORD\|REDACTED_PASSWORD\|REDACTED_WATCHTOWER_TOKEN" . 2>/dev/null
grep -r "2e80b1b7d3a\|eca299ae59\|rxmr4tJoqfu\|ZjCofRlfm6\|QE5SudhZ99" . 2>/dev/null
# All should return no results

# 4. Verify GitOps deployments are healthy in Portainer:
#    https://192.168.0.200:9443
```

---

## Going Forward — Preventing This Again

The root cause: secrets hard-coded in compose files that get committed to git.

**Rules:**
1. **Never hard-code secrets in compose files** — use Docker Secrets, or an
   `.env` file excluded by `.gitignore` (Portainer can load env files from the
   host at deploy time)
2. **Never put real values in documentation** — use `YOUR_API_KEY` placeholders
3. **Never create `Secrets.txt` or `CREDENTIALS.md` files in the repo** — use
   a password manager (you already have Vaultwarden/Bitwarden)
4. **Run the sanitizer locally** before any commit that touches secrets:

```bash
# Test in a temp copy — see what the sanitizer would catch:
tmpdir=$(mktemp -d)
cp -r /path/to/homelab "$tmpdir/"
python3 "$tmpdir/homelab/.gitea/sanitize.py"
```

## Related Documentation

- [Security Hardening](../security/SERVER_HARDENING.md)
- [Repository Sanitization](../admin/REPOSITORY_SANITIZATION.md)
- [GitOps Deployment Guide](../admin/GITOPS_DEPLOYMENT_GUIDE.md)

## Portainer Git Credential Rotation

The saved Git credential **`portainer-homelab`** (credId: 1) is used by ~43 stacks to
pull compose files from `git.vish.gg`. When the Gitea token expires or is rotated,
all those stacks fail to redeploy.

```bash
# 1. Generate a new Gitea token at https://git.vish.gg/user/settings/applications
#    Scope: read:repository

# 2. Test the token:
curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: token YOUR_NEW_TOKEN" \
  "https://git.vish.gg/api/v1/repos/Vish/homelab"
# Should return 200

# 3. Update in Portainer:
curl -k -s -X PUT \
  -H "X-API-Key: "REDACTED_API_KEY" \
  -H "Content-Type: application/json" \
  "https://192.168.0.200:9443/api/users/1/gitcredentials/1" \
  -d '{"name":"portainer-homelab","username":"vish","password":"YOUR_NEW_TOKEN"}'
```

> Note: The API update may not immediately propagate to automated pulls.
> Pass credentials inline in redeploy calls to force use of the new token.

---

## Change Log

- 2026-02-27 — Incident: sanitization commit `037d766a` replaced credentials with
  `REDACTED_PASSWORD` placeholders across 14 compose files. All affected containers
  detected via Portainer API env scan and restored from `git show 037d766a^`. Added
  Portainer Git credential rotation section above.
- 2026-02-20 — Initial creation (8 items)
- 2026-02-20 — Expanded after full private repo audit (27 items across 34 exposure categories)