homelab-optimized/docs/runbooks/certificate-renewal.md

# SSL/TLS Certificate Renewal Runbook

## Overview
This runbook covers SSL/TLS certificate management across the homelab, including Let's Encrypt certificates, Cloudflare Origin certificates, and self-signed certificates. It provides procedures for manual renewal, troubleshooting auto-renewal, and emergency certificate fixes.

## Prerequisites
- [ ] SSH access to relevant hosts
- [ ] Cloudflare account access (if using Cloudflare)
- [ ] Domain DNS control
- [ ] Root/sudo privileges on hosts
- [ ] Backup of current certificates

## Metadata
- **Estimated Time**: 15-45 minutes
- **Risk Level**: Medium (service downtime if misconfigured)
- **Requires Downtime**: Minimal (few seconds during reload)
- **Reversible**: Yes (can restore old certificates)
- **Tested On**: 2026-02-14

## Certificate Types in Homelab

| Type | Used For | Renewal Method | Expiration |
|------|----------|----------------|------------|
| **Let's Encrypt** | Public-facing services | Certbot auto-renewal | 90 days |
| **Cloudflare Origin** | Services behind Cloudflare Tunnel | Manual/Cloudflare dashboard | 15 years |
| **Synology Certificates** | Synology DSM, services | Synology DSM auto-renewal | 90 days |
| **Self-Signed** | Internal/dev services | Manual generation | As configured |

## Certificate Inventory

Document your current certificates:

```bash
# Check Let's Encrypt certificates (on Linux hosts)
sudo certbot certificates

# Check Synology certificates
# DSM UI → Control Panel → Security → Certificate
# Or SSH:
sudo cat /usr/syno/etc/certificate/_archive/*/cert.pem | openssl x509 -text -noout

# Check certificate expiration for any domain
echo | openssl s_client -servername service.vish.gg -connect service.vish.gg:443 2>/dev/null | openssl x509 -noout -dates

# Check all certificates at once
for domain in st.vish.gg gf.vish.gg mx.vish.gg; do
    echo "=== $domain ==="
    echo | timeout 5 openssl s_client -servername $domain -connect $domain:443 2>/dev/null | openssl x509 -noout -dates
    echo
done
```

Create inventory:
```markdown
| Domain | Type | Expiry Date | Auto-Renew | Status |
|--------|------|-------------|------------|--------|
| vish.gg | Let's Encrypt | 2026-05-15 | ✅ Yes | ✅ Valid |
| st.vish.gg | Let's Encrypt | 2026-05-15 | ✅ Yes | ✅ Valid |
| gf.vish.gg | Let's Encrypt | 2026-05-15 | ✅ Yes | ✅ Valid |
```

## Let's Encrypt Certificate Renewal

### Automatic Renewal (Certbot)

Let's Encrypt certificates should auto-renew. Check the renewal setup:

```bash
# Check certbot timer status (systemd)
sudo systemctl status certbot.timer

# Check cron job (if using cron)
sudo crontab -l | grep certbot

# Test renewal (dry-run, doesn't actually renew)
sudo certbot renew --dry-run

# Expected output:
# Congratulations, all simulated renewals succeeded
```

### Manual Renewal

If auto-renewal fails or you need to renew manually:

```bash
# Renew all certificates
sudo certbot renew

# Renew specific certificate
sudo certbot renew --cert-name vish.gg

# Force renewal (even if not expired)
sudo certbot renew --force-renewal

# Renew with verbose output for troubleshooting
sudo certbot renew --verbose
```

After renewal, reload web servers:

```bash
# Nginx
sudo nginx -t  # Test configuration
sudo systemctl reload nginx

# Apache
sudo apachectl configtest
sudo systemctl reload apache2
```

### Let's Encrypt with Nginx Proxy Manager

If using Nginx Proxy Manager (NPM):

1. Open NPM UI (typically port 81)
2. Go to **SSL Certificates** tab
3. Certificates should auto-renew 30 days before expiry
4. To force renewal:
   - Click the certificate
   - Click **Renew** button
5. No service reload needed (NPM handles it)

## Synology Certificate Renewal

### Automatic Renewal on Synology NAS

```bash
# SSH to Synology NAS (Atlantis or Calypso)
ssh atlantis  # or calypso

# Check certificate status
sudo /usr/syno/sbin/syno-letsencrypt list

# Force renewal check
sudo /usr/syno/sbin/syno-letsencrypt renew-all

# Check renewal logs
sudo cat /var/log/letsencrypt/letsencrypt.log

# Verify certificate expiry
sudo openssl x509 -in /usr/syno/etc/certificate/system/default/cert.pem -text -noout | grep "Not After"
```

### Via Synology DSM UI

1. Log in to DSM
2. **Control Panel** → **Security** → **Certificate**
3. Select certificate → Click **Renew**
4. DSM will automatically renew and apply
5. No manual reload needed

### Synology Certificate Configuration

Enable auto-renewal in DSM:
1. **Control Panel** → **Security** → **Certificate**
2. Click **Settings** button
3. Check **Auto-renew certificate**
4. Synology will renew 30 days before expiry

## Stoatchat Certificates (Gaming VPS)

The Stoatchat gaming server uses Let's Encrypt with Certbot:

```bash
# SSH to gaming VPS
ssh root@gaming-vps

# Check certificates
sudo certbot certificates

# Domains covered:
# - st.vish.gg
# - api.st.vish.gg
# - events.st.vish.gg
# - files.st.vish.gg
# - proxy.st.vish.gg
# - voice.st.vish.gg

# Renew all
sudo certbot renew

# Reload Nginx
sudo systemctl reload nginx
```

Auto-renewal cron:
```bash
# Check certbot timer
sudo systemctl status certbot.timer

# Or check cron
sudo crontab -l | grep certbot
```

## Cloudflare Origin Certificates

For services using Cloudflare Tunnel:

### Generate New Origin Certificate

1. Log in to Cloudflare Dashboard
2. Select domain (vish.gg)
3. **SSL/TLS** → **Origin Server**
4. Click **Create Certificate**
5. Configure:
   - **Private key type**: RSA (2048)
   - **Hostnames**: *.vish.gg, vish.gg
   - **Certificate validity**: 15 years
6. Copy certificate and private key
7. Save to secure location

### Install Origin Certificate

```bash
# SSH to target host
ssh [host]

# Create certificate files
sudo nano /etc/ssl/cloudflare/cert.pem
# Paste certificate

sudo nano /etc/ssl/cloudflare/key.pem
# Paste private key

# Set permissions
sudo chmod 644 /etc/ssl/cloudflare/cert.pem
sudo chmod 600 /etc/ssl/cloudflare/key.pem

# Update Nginx configuration
sudo nano /etc/nginx/sites-available/[service]

# Use new certificate
ssl_certificate /etc/ssl/cloudflare/cert.pem;
ssl_certificate_key /etc/ssl/cloudflare/key.pem;

# Test and reload
sudo nginx -t
sudo systemctl reload nginx
```

## Self-Signed Certificates (Internal/Dev)

For internal-only services not exposed publicly:

### Generate Self-Signed Certificate

```bash
# Generate 10-year self-signed certificate
sudo openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
    -keyout /etc/ssl/private/selfsigned.key \
    -out /etc/ssl/certs/selfsigned.crt \
    -subj "/C=US/ST=State/L=City/O=Homelab/CN=internal.vish.local"

# Generate with SAN (Subject Alternative Names) for multiple domains
sudo openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
    -keyout /etc/ssl/private/selfsigned.key \
    -out /etc/ssl/certs/selfsigned.crt \
    -subj "/C=US/ST=State/L=City/O=Homelab/CN=*.vish.local" \
    -addext "subjectAltName=DNS:*.vish.local,DNS:vish.local"

# Set permissions
sudo chmod 600 /etc/ssl/private/selfsigned.key
sudo chmod 644 /etc/ssl/certs/selfsigned.crt
```

### Install in Services

Update Docker Compose to mount certificates:

```yaml
services:
  service:
    volumes:
      - /etc/ssl/certs/selfsigned.crt:/etc/ssl/certs/cert.pem:ro
      - /etc/ssl/private/selfsigned.key:/etc/ssl/private/key.pem:ro
```

## Monitoring Certificate Expiration

### Set Up Expiration Alerts

Create a certificate monitoring script:

```bash
sudo nano /usr/local/bin/check-certificates.sh
```

```bash
#!/bin/bash
# Certificate Expiration Monitoring Script

DOMAINS=(
    "vish.gg"
    "st.vish.gg"
    "gf.vish.gg"
    "mx.vish.gg"
)

ALERT_DAYS=30  # Alert if expiring within 30 days
WEBHOOK_URL="https://ntfy.sh/REDACTED_TOPIC"  # Your notification webhook

for domain in "${DOMAINS[@]}"; do
    echo "Checking $domain..."

    # Get certificate expiration date
    expiry=$(echo | openssl s_client -servername $domain -connect $domain:443 2>/dev/null | \
             openssl x509 -noout -dates | grep "notAfter" | cut -d= -f2)

    # Convert to epoch time
    expiry_epoch=$(date -d "$expiry" +%s)
    current_epoch=$(date +%s)
    days_left=$(( ($expiry_epoch - $current_epoch) / 86400 ))

    echo "$domain expires in $days_left days"

    if [ $days_left -lt $ALERT_DAYS ]; then
        # Send alert
        curl -H "Title: Certificate Expiring Soon" \
             -H "Priority: high" \
             -H "Tags: warning,certificate" \
             -d "Certificate for $domain expires in $days_left days!" \
             $WEBHOOK_URL

        echo "⚠️  Alert sent for $domain"
    fi
    echo
done
```

Make executable and add to cron:
```bash
sudo chmod +x /usr/local/bin/check-certificates.sh

# Add to cron (daily at 9 AM)
(crontab -l 2>/dev/null; echo "0 9 * * * /usr/local/bin/check-certificates.sh") | crontab -
```

### Grafana Dashboard

Add certificate monitoring to Grafana:

```bash
# Install blackbox_exporter for HTTPS probing
# Add to prometheus.yml:

scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://vish.gg
        - https://st.vish.gg
        - https://gf.vish.gg
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

# Create alert rule:
- alert: SSLCertificateExpiring
  expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
  labels:
    severity: warning
  annotations:
    summary: "SSL certificate expiring soon"
    description: "SSL certificate for {{ $labels.instance }} expires in {{ $value | REDACTED_APP_PASSWORD }}"
```

## Troubleshooting

### Issue: Certbot Renewal Failing

**Symptoms**: `certbot renew` fails with DNS or HTTP challenge errors

**Solutions**:

```bash
# Check detailed error logs
sudo certbot renew --verbose

# Common issues:

# 1. Port 80/443 not accessible
sudo ufw status  # Check firewall
sudo netstat -tlnp | grep :80  # Check if port is listening

# 2. DNS not resolving correctly
dig vish.gg  # Verify DNS points to correct IP

# 3. Rate limits hit
# Let's Encrypt has rate limits: 50 certificates per domain per week
# Wait 7 days or use --staging for testing

# 4. Webroot path incorrect
sudo certbot renew --webroot -w /var/www/html

# 5. Try force renewal with different challenge
sudo certbot renew --force-renewal --preferred-challenges dns
```

### Issue: Certificate Valid But Browser Shows Warning

**Symptoms**: Certificate is valid but browsers show security warning

**Solutions**:

```bash
# Check certificate chain
openssl s_client -connect vish.gg:443 -showcerts

# Ensure intermediate certificates are included
# Nginx: Use fullchain.pem, not cert.pem
ssl_certificate /etc/letsencrypt/live/vish.gg/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/vish.gg/privkey.pem;

# Test SSL configuration
curl -I https://vish.gg
# Or use: https://www.ssllabs.com/ssltest/
```

### Issue: Synology Certificate Not Auto-Renewing

**Symptoms**: DSM certificate expired or shows renewal error

**Solutions**:

```bash
# SSH to Synology
ssh atlantis

# Check renewal logs
sudo cat /var/log/letsencrypt/letsencrypt.log

# Common issues:

# 1. Port 80 forwarding
# Ensure port 80 is forwarded to NAS during renewal

# 2. Domain validation
# Check DNS points to correct external IP

# 3. Force renewal
sudo /usr/syno/sbin/syno-letsencrypt renew-all

# 4. Restart certificate service
sudo synosystemctl restart nginx
```

### Issue: Nginx Won't Reload After Certificate Update

**Symptoms**: `nginx -t` shows SSL errors

**Solutions**:

```bash
# Test Nginx configuration
sudo nginx -t

# Common errors:

# 1. Certificate path incorrect
# Fix: Update nginx config with correct path

# 2. Certificate and key mismatch
# Verify:
sudo openssl x509 -noout -modulus -in cert.pem | openssl md5
sudo openssl rsa -noout -modulus -in key.pem | openssl md5
# MD5 sums should match

# 3. Permission issues
sudo chmod 644 /etc/ssl/certs/cert.pem
sudo chmod 600 /etc/ssl/private/key.pem
sudo chown root:root /etc/ssl/certs/cert.pem /etc/ssl/private/key.pem

# 4. SELinux blocking (if enabled)
sudo setsebool -P httpd_read_user_content 1
```

## Emergency Certificate Fix

If a certificate expires and services are down:

### Quick Fix: Use Self-Signed Temporarily

```bash
# Generate emergency self-signed certificate
sudo openssl req -x509 -nodes -days 30 -newkey rsa:2048 \
    -keyout /tmp/emergency.key \
    -out /tmp/emergency.crt \
    -subj "/CN=*.vish.gg"

# Update Nginx to use emergency cert
sudo nano /etc/nginx/sites-available/default

ssl_certificate /tmp/emergency.crt;
ssl_certificate_key /tmp/emergency.key;

# Reload Nginx
sudo nginx -t && sudo systemctl reload nginx

# Services are now accessible (with browser warning)
# Then fix proper certificate renewal
```

### Restore from Backup

```bash
# If certificates were backed up
sudo cp /backup/letsencrypt/archive/vish.gg/* /etc/letsencrypt/archive/vish.gg/

# Update symlinks
sudo certbot certificates  # Shows current status
sudo certbot install --cert-name vish.gg
```

## Best Practices

### Renewal Schedule
- Let's Encrypt certificates renew at 60 days (30 days before expiry)
- Check certificates monthly
- Set up expiration alerts
- Test renewal process quarterly

### Backup Certificates
```bash
# Backup Let's Encrypt certificates
sudo tar czf ~/letsencrypt-backup-$(date +%Y%m%d).tar.gz /etc/letsencrypt/

# Backup Synology certificates
# Done via Synology backup tasks

# Store backups securely (encrypted, off-site)
```

### Documentation
- Document which certificates are used where
- Keep inventory of expiration dates
- Document renewal procedures
- Note any special configurations

## Verification Checklist

After certificate renewal:

- [ ] Certificate renewed successfully
- [ ] Certificate expiry date extended
- [ ] Web servers reloaded without errors
- [ ] All services accessible via HTTPS
- [ ] No browser security warnings
- [ ] Certificate chain complete
- [ ] Auto-renewal still enabled
- [ ] Monitoring updated (if needed)

## Related Documentation

- [Infrastructure Overview](../infrastructure/INFRASTRUCTURE_OVERVIEW.md)
- [Nginx Configuration](../infrastructure/networking.md)
- [Cloudflare Tunnels Setup](../infrastructure/cloudflare-tunnels-setup.md)
- [Emergency Access Guide](../troubleshooting/EMERGENCY_ACCESS_GUIDE.md)

## Change Log

- 2026-02-14 - Initial creation
- 2026-02-14 - Added monitoring and troubleshooting sections