571 lines
14 KiB
Markdown
571 lines
14 KiB
Markdown
# SSL/TLS Certificate Renewal Runbook
|
|
|
|
## Overview
|
|
This runbook covers SSL/TLS certificate management across the homelab, including Let's Encrypt certificates, Cloudflare Origin certificates, and self-signed certificates. It provides procedures for manual renewal, troubleshooting auto-renewal, and emergency certificate fixes.
|
|
|
|
## Prerequisites
|
|
- [ ] SSH access to relevant hosts
|
|
- [ ] Cloudflare account access (if using Cloudflare)
|
|
- [ ] Domain DNS control
|
|
- [ ] Root/sudo privileges on hosts
|
|
- [ ] Backup of current certificates
|
|
|
|
## Metadata
|
|
- **Estimated Time**: 15-45 minutes
|
|
- **Risk Level**: Medium (service downtime if misconfigured)
|
|
- **Requires Downtime**: Minimal (few seconds during reload)
|
|
- **Reversible**: Yes (can restore old certificates)
|
|
- **Tested On**: 2026-02-14
|
|
|
|
## Certificate Types in Homelab
|
|
|
|
| Type | Used For | Renewal Method | Expiration |
|
|
|------|----------|----------------|------------|
|
|
| **Let's Encrypt** | Public-facing services | Certbot auto-renewal | 90 days |
|
|
| **Cloudflare Origin** | Services behind Cloudflare Tunnel | Manual/Cloudflare dashboard | 15 years |
|
|
| **Synology Certificates** | Synology DSM, services | Synology DSM auto-renewal | 90 days |
|
|
| **Self-Signed** | Internal/dev services | Manual generation | As configured |
|
|
|
|
## Certificate Inventory
|
|
|
|
Document your current certificates:
|
|
|
|
```bash
|
|
# Check Let's Encrypt certificates (on Linux hosts)
|
|
sudo certbot certificates
|
|
|
|
# Check Synology certificates
|
|
# DSM UI → Control Panel → Security → Certificate
|
|
# Or SSH:
|
|
sudo cat /usr/syno/etc/certificate/_archive/*/cert.pem | openssl x509 -text -noout
|
|
|
|
# Check certificate expiration for any domain
|
|
echo | openssl s_client -servername service.vish.gg -connect service.vish.gg:443 2>/dev/null | openssl x509 -noout -dates
|
|
|
|
# Check all certificates at once
|
|
for domain in st.vish.gg gf.vish.gg mx.vish.gg; do
|
|
echo "=== $domain ==="
|
|
echo | timeout 5 openssl s_client -servername $domain -connect $domain:443 2>/dev/null | openssl x509 -noout -dates
|
|
echo
|
|
done
|
|
```
|
|
|
|
Create inventory:
|
|
```markdown
|
|
| Domain | Type | Expiry Date | Auto-Renew | Status |
|
|
|--------|------|-------------|------------|--------|
|
|
| vish.gg | Let's Encrypt | 2026-05-15 | ✅ Yes | ✅ Valid |
|
|
| st.vish.gg | Let's Encrypt | 2026-05-15 | ✅ Yes | ✅ Valid |
|
|
| gf.vish.gg | Let's Encrypt | 2026-05-15 | ✅ Yes | ✅ Valid |
|
|
```
|
|
|
|
## Let's Encrypt Certificate Renewal
|
|
|
|
### Automatic Renewal (Certbot)
|
|
|
|
Let's Encrypt certificates should auto-renew. Check the renewal setup:
|
|
|
|
```bash
|
|
# Check certbot timer status (systemd)
|
|
sudo systemctl status certbot.timer
|
|
|
|
# Check cron job (if using cron)
|
|
sudo crontab -l | grep certbot
|
|
|
|
# Test renewal (dry-run, doesn't actually renew)
|
|
sudo certbot renew --dry-run
|
|
|
|
# Expected output:
|
|
# Congratulations, all simulated renewals succeeded
|
|
```
|
|
|
|
### Manual Renewal
|
|
|
|
If auto-renewal fails or you need to renew manually:
|
|
|
|
```bash
|
|
# Renew all certificates
|
|
sudo certbot renew
|
|
|
|
# Renew specific certificate
|
|
sudo certbot renew --cert-name vish.gg
|
|
|
|
# Force renewal (even if not expired)
|
|
sudo certbot renew --force-renewal
|
|
|
|
# Renew with verbose output for troubleshooting
|
|
sudo certbot renew --verbose
|
|
```
|
|
|
|
After renewal, reload web servers:
|
|
|
|
```bash
|
|
# Nginx
|
|
sudo nginx -t # Test configuration
|
|
sudo systemctl reload nginx
|
|
|
|
# Apache
|
|
sudo apachectl configtest
|
|
sudo systemctl reload apache2
|
|
```
|
|
|
|
### Let's Encrypt with Nginx Proxy Manager
|
|
|
|
If using Nginx Proxy Manager (NPM):
|
|
|
|
1. Open NPM UI (typically port 81)
|
|
2. Go to **SSL Certificates** tab
|
|
3. Certificates should auto-renew 30 days before expiry
|
|
4. To force renewal:
|
|
- Click the certificate
|
|
- Click **Renew** button
|
|
5. No service reload needed (NPM handles it)
|
|
|
|
## Synology Certificate Renewal
|
|
|
|
### Automatic Renewal on Synology NAS
|
|
|
|
```bash
|
|
# SSH to Synology NAS (Atlantis or Calypso)
|
|
ssh atlantis # or calypso
|
|
|
|
# Check certificate status
|
|
sudo /usr/syno/sbin/syno-letsencrypt list
|
|
|
|
# Force renewal check
|
|
sudo /usr/syno/sbin/syno-letsencrypt renew-all
|
|
|
|
# Check renewal logs
|
|
sudo cat /var/log/letsencrypt/letsencrypt.log
|
|
|
|
# Verify certificate expiry
|
|
sudo openssl x509 -in /usr/syno/etc/certificate/system/default/cert.pem -text -noout | grep "Not After"
|
|
```
|
|
|
|
### Via Synology DSM UI
|
|
|
|
1. Log in to DSM
|
|
2. **Control Panel** → **Security** → **Certificate**
|
|
3. Select certificate → Click **Renew**
|
|
4. DSM will automatically renew and apply
|
|
5. No manual reload needed
|
|
|
|
### Synology Certificate Configuration
|
|
|
|
Enable auto-renewal in DSM:
|
|
1. **Control Panel** → **Security** → **Certificate**
|
|
2. Click **Settings** button
|
|
3. Check **Auto-renew certificate**
|
|
4. Synology will renew 30 days before expiry
|
|
|
|
## Stoatchat Certificates (Gaming VPS)
|
|
|
|
The Stoatchat gaming server uses Let's Encrypt with Certbot:
|
|
|
|
```bash
|
|
# SSH to gaming VPS
|
|
ssh root@gaming-vps
|
|
|
|
# Check certificates
|
|
sudo certbot certificates
|
|
|
|
# Domains covered:
|
|
# - st.vish.gg
|
|
# - api.st.vish.gg
|
|
# - events.st.vish.gg
|
|
# - files.st.vish.gg
|
|
# - proxy.st.vish.gg
|
|
# - voice.st.vish.gg
|
|
|
|
# Renew all
|
|
sudo certbot renew
|
|
|
|
# Reload Nginx
|
|
sudo systemctl reload nginx
|
|
```
|
|
|
|
Auto-renewal cron:
|
|
```bash
|
|
# Check certbot timer
|
|
sudo systemctl status certbot.timer
|
|
|
|
# Or check cron
|
|
sudo crontab -l | grep certbot
|
|
```
|
|
|
|
## Cloudflare Origin Certificates
|
|
|
|
For services using Cloudflare Tunnel:
|
|
|
|
### Generate New Origin Certificate
|
|
|
|
1. Log in to Cloudflare Dashboard
|
|
2. Select domain (vish.gg)
|
|
3. **SSL/TLS** → **Origin Server**
|
|
4. Click **Create Certificate**
|
|
5. Configure:
|
|
- **Private key type**: RSA (2048)
|
|
- **Hostnames**: *.vish.gg, vish.gg
|
|
- **Certificate validity**: 15 years
|
|
6. Copy certificate and private key
|
|
7. Save to secure location
|
|
|
|
### Install Origin Certificate
|
|
|
|
```bash
|
|
# SSH to target host
|
|
ssh [host]
|
|
|
|
# Create certificate files
|
|
sudo nano /etc/ssl/cloudflare/cert.pem
|
|
# Paste certificate
|
|
|
|
sudo nano /etc/ssl/cloudflare/key.pem
|
|
# Paste private key
|
|
|
|
# Set permissions
|
|
sudo chmod 644 /etc/ssl/cloudflare/cert.pem
|
|
sudo chmod 600 /etc/ssl/cloudflare/key.pem
|
|
|
|
# Update Nginx configuration
|
|
sudo nano /etc/nginx/sites-available/[service]
|
|
|
|
# Use new certificate
|
|
ssl_certificate /etc/ssl/cloudflare/cert.pem;
|
|
ssl_certificate_key /etc/ssl/cloudflare/key.pem;
|
|
|
|
# Test and reload
|
|
sudo nginx -t
|
|
sudo systemctl reload nginx
|
|
```
|
|
|
|
## Self-Signed Certificates (Internal/Dev)
|
|
|
|
For internal-only services not exposed publicly:
|
|
|
|
### Generate Self-Signed Certificate
|
|
|
|
```bash
|
|
# Generate 10-year self-signed certificate
|
|
sudo openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
|
|
-keyout /etc/ssl/private/selfsigned.key \
|
|
-out /etc/ssl/certs/selfsigned.crt \
|
|
-subj "/C=US/ST=State/L=City/O=Homelab/CN=internal.vish.local"
|
|
|
|
# Generate with SAN (Subject Alternative Names) for multiple domains
|
|
sudo openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
|
|
-keyout /etc/ssl/private/selfsigned.key \
|
|
-out /etc/ssl/certs/selfsigned.crt \
|
|
-subj "/C=US/ST=State/L=City/O=Homelab/CN=*.vish.local" \
|
|
-addext "subjectAltName=DNS:*.vish.local,DNS:vish.local"
|
|
|
|
# Set permissions
|
|
sudo chmod 600 /etc/ssl/private/selfsigned.key
|
|
sudo chmod 644 /etc/ssl/certs/selfsigned.crt
|
|
```
|
|
|
|
### Install in Services
|
|
|
|
Update Docker Compose to mount certificates:
|
|
|
|
```yaml
|
|
services:
|
|
service:
|
|
volumes:
|
|
- /etc/ssl/certs/selfsigned.crt:/etc/ssl/certs/cert.pem:ro
|
|
- /etc/ssl/private/selfsigned.key:/etc/ssl/private/key.pem:ro
|
|
```
|
|
|
|
## Monitoring Certificate Expiration
|
|
|
|
### Set Up Expiration Alerts
|
|
|
|
Create a certificate monitoring script:
|
|
|
|
```bash
|
|
sudo nano /usr/local/bin/check-certificates.sh
|
|
```
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# Certificate Expiration Monitoring Script
|
|
|
|
DOMAINS=(
|
|
"vish.gg"
|
|
"st.vish.gg"
|
|
"gf.vish.gg"
|
|
"mx.vish.gg"
|
|
)
|
|
|
|
ALERT_DAYS=30 # Alert if expiring within 30 days
|
|
WEBHOOK_URL="https://ntfy.sh/REDACTED_TOPIC" # Your notification webhook
|
|
|
|
for domain in "${DOMAINS[@]}"; do
|
|
echo "Checking $domain..."
|
|
|
|
# Get certificate expiration date
|
|
expiry=$(echo | openssl s_client -servername $domain -connect $domain:443 2>/dev/null | \
|
|
openssl x509 -noout -dates | grep "notAfter" | cut -d= -f2)
|
|
|
|
# Convert to epoch time
|
|
expiry_epoch=$(date -d "$expiry" +%s)
|
|
current_epoch=$(date +%s)
|
|
days_left=$(( ($expiry_epoch - $current_epoch) / 86400 ))
|
|
|
|
echo "$domain expires in $days_left days"
|
|
|
|
if [ $days_left -lt $ALERT_DAYS ]; then
|
|
# Send alert
|
|
curl -H "Title: Certificate Expiring Soon" \
|
|
-H "Priority: high" \
|
|
-H "Tags: warning,certificate" \
|
|
-d "Certificate for $domain expires in $days_left days!" \
|
|
$WEBHOOK_URL
|
|
|
|
echo "⚠️ Alert sent for $domain"
|
|
fi
|
|
echo
|
|
done
|
|
```
|
|
|
|
Make executable and add to cron:
|
|
```bash
|
|
sudo chmod +x /usr/local/bin/check-certificates.sh
|
|
|
|
# Add to cron (daily at 9 AM)
|
|
(crontab -l 2>/dev/null; echo "0 9 * * * /usr/local/bin/check-certificates.sh") | crontab -
|
|
```
|
|
|
|
### Grafana Dashboard
|
|
|
|
Add certificate monitoring to Grafana:
|
|
|
|
```bash
|
|
# Install blackbox_exporter for HTTPS probing
|
|
# Add to prometheus.yml:
|
|
|
|
scrape_configs:
|
|
- job_name: 'blackbox'
|
|
metrics_path: /probe
|
|
params:
|
|
module: [http_2xx]
|
|
static_configs:
|
|
- targets:
|
|
- https://vish.gg
|
|
- https://st.vish.gg
|
|
- https://gf.vish.gg
|
|
relabel_configs:
|
|
- source_labels: [__address__]
|
|
target_label: __param_target
|
|
- source_labels: [__param_target]
|
|
target_label: instance
|
|
- target_label: __address__
|
|
replacement: blackbox-exporter:9115
|
|
|
|
# Create alert rule:
|
|
- alert: SSLCertificateExpiring
|
|
expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "SSL certificate expiring soon"
|
|
description: "SSL certificate for {{ $labels.instance }} expires in {{ $value | REDACTED_APP_PASSWORD }}"
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Issue: Certbot Renewal Failing
|
|
|
|
**Symptoms**: `certbot renew` fails with DNS or HTTP challenge errors
|
|
|
|
**Solutions**:
|
|
|
|
```bash
|
|
# Check detailed error logs
|
|
sudo certbot renew --verbose
|
|
|
|
# Common issues:
|
|
|
|
# 1. Port 80/443 not accessible
|
|
sudo ufw status # Check firewall
|
|
sudo netstat -tlnp | grep :80 # Check if port is listening
|
|
|
|
# 2. DNS not resolving correctly
|
|
dig vish.gg # Verify DNS points to correct IP
|
|
|
|
# 3. Rate limits hit
|
|
# Let's Encrypt has rate limits: 50 certificates per domain per week
|
|
# Wait 7 days or use --staging for testing
|
|
|
|
# 4. Webroot path incorrect
|
|
sudo certbot renew --webroot -w /var/www/html
|
|
|
|
# 5. Try force renewal with different challenge
|
|
sudo certbot renew --force-renewal --preferred-challenges dns
|
|
```
|
|
|
|
### Issue: Certificate Valid But Browser Shows Warning
|
|
|
|
**Symptoms**: Certificate is valid but browsers show security warning
|
|
|
|
**Solutions**:
|
|
|
|
```bash
|
|
# Check certificate chain
|
|
openssl s_client -connect vish.gg:443 -showcerts
|
|
|
|
# Ensure intermediate certificates are included
|
|
# Nginx: Use fullchain.pem, not cert.pem
|
|
ssl_certificate /etc/letsencrypt/live/vish.gg/fullchain.pem;
|
|
ssl_certificate_key /etc/letsencrypt/live/vish.gg/privkey.pem;
|
|
|
|
# Test SSL configuration
|
|
curl -I https://vish.gg
|
|
# Or use: https://www.ssllabs.com/ssltest/
|
|
```
|
|
|
|
### Issue: Synology Certificate Not Auto-Renewing
|
|
|
|
**Symptoms**: DSM certificate expired or shows renewal error
|
|
|
|
**Solutions**:
|
|
|
|
```bash
|
|
# SSH to Synology
|
|
ssh atlantis
|
|
|
|
# Check renewal logs
|
|
sudo cat /var/log/letsencrypt/letsencrypt.log
|
|
|
|
# Common issues:
|
|
|
|
# 1. Port 80 forwarding
|
|
# Ensure port 80 is forwarded to NAS during renewal
|
|
|
|
# 2. Domain validation
|
|
# Check DNS points to correct external IP
|
|
|
|
# 3. Force renewal
|
|
sudo /usr/syno/sbin/syno-letsencrypt renew-all
|
|
|
|
# 4. Restart certificate service
|
|
sudo synosystemctl restart nginx
|
|
```
|
|
|
|
### Issue: Nginx Won't Reload After Certificate Update
|
|
|
|
**Symptoms**: `nginx -t` shows SSL errors
|
|
|
|
**Solutions**:
|
|
|
|
```bash
|
|
# Test Nginx configuration
|
|
sudo nginx -t
|
|
|
|
# Common errors:
|
|
|
|
# 1. Certificate path incorrect
|
|
# Fix: Update nginx config with correct path
|
|
|
|
# 2. Certificate and key mismatch
|
|
# Verify:
|
|
sudo openssl x509 -noout -modulus -in cert.pem | openssl md5
|
|
sudo openssl rsa -noout -modulus -in key.pem | openssl md5
|
|
# MD5 sums should match
|
|
|
|
# 3. Permission issues
|
|
sudo chmod 644 /etc/ssl/certs/cert.pem
|
|
sudo chmod 600 /etc/ssl/private/key.pem
|
|
sudo chown root:root /etc/ssl/certs/cert.pem /etc/ssl/private/key.pem
|
|
|
|
# 4. SELinux blocking (if enabled)
|
|
sudo setsebool -P httpd_read_user_content 1
|
|
```
|
|
|
|
## Emergency Certificate Fix
|
|
|
|
If a certificate expires and services are down:
|
|
|
|
### Quick Fix: Use Self-Signed Temporarily
|
|
|
|
```bash
|
|
# Generate emergency self-signed certificate
|
|
sudo openssl req -x509 -nodes -days 30 -newkey rsa:2048 \
|
|
-keyout /tmp/emergency.key \
|
|
-out /tmp/emergency.crt \
|
|
-subj "/CN=*.vish.gg"
|
|
|
|
# Update Nginx to use emergency cert
|
|
sudo nano /etc/nginx/sites-available/default
|
|
|
|
ssl_certificate /tmp/emergency.crt;
|
|
ssl_certificate_key /tmp/emergency.key;
|
|
|
|
# Reload Nginx
|
|
sudo nginx -t && sudo systemctl reload nginx
|
|
|
|
# Services are now accessible (with browser warning)
|
|
# Then fix proper certificate renewal
|
|
```
|
|
|
|
### Restore from Backup
|
|
|
|
```bash
|
|
# If certificates were backed up
|
|
sudo cp /backup/letsencrypt/archive/vish.gg/* /etc/letsencrypt/archive/vish.gg/
|
|
|
|
# Update symlinks
|
|
sudo certbot certificates # Shows current status
|
|
sudo certbot install --cert-name vish.gg
|
|
```
|
|
|
|
## Best Practices
|
|
|
|
### Renewal Schedule
|
|
- Let's Encrypt certificates renew at 60 days (30 days before expiry)
|
|
- Check certificates monthly
|
|
- Set up expiration alerts
|
|
- Test renewal process quarterly
|
|
|
|
### Backup Certificates
|
|
```bash
|
|
# Backup Let's Encrypt certificates
|
|
sudo tar czf ~/letsencrypt-backup-$(date +%Y%m%d).tar.gz /etc/letsencrypt/
|
|
|
|
# Backup Synology certificates
|
|
# Done via Synology backup tasks
|
|
|
|
# Store backups securely (encrypted, off-site)
|
|
```
|
|
|
|
### Documentation
|
|
- Document which certificates are used where
|
|
- Keep inventory of expiration dates
|
|
- Document renewal procedures
|
|
- Note any special configurations
|
|
|
|
## Verification Checklist
|
|
|
|
After certificate renewal:
|
|
|
|
- [ ] Certificate renewed successfully
|
|
- [ ] Certificate expiry date extended
|
|
- [ ] Web servers reloaded without errors
|
|
- [ ] All services accessible via HTTPS
|
|
- [ ] No browser security warnings
|
|
- [ ] Certificate chain complete
|
|
- [ ] Auto-renewal still enabled
|
|
- [ ] Monitoring updated (if needed)
|
|
|
|
## Related Documentation
|
|
|
|
- [Infrastructure Overview](../infrastructure/INFRASTRUCTURE_OVERVIEW.md)
|
|
- [Nginx Configuration](../infrastructure/networking.md)
|
|
- [Cloudflare Tunnels Setup](../infrastructure/cloudflare-tunnels-setup.md)
|
|
- [Emergency Access Guide](../troubleshooting/EMERGENCY_ACCESS_GUIDE.md)
|
|
|
|
## Change Log
|
|
|
|
- 2026-02-14 - Initial creation
|
|
- 2026-02-14 - Added monitoring and troubleshooting sections
|