Sanitized mirror from private repository - 2026-04-19 08:15:48 UTC
This commit is contained in:
Gitea Mirror Bot
570
docs/runbooks/certificate-renewal.md
Normal file
570
docs/runbooks/certificate-renewal.md
Normal file
@@ -0,0 +1,570 @@
|
||||
# SSL/TLS Certificate Renewal Runbook
|
||||
|
||||
## Overview
|
||||
This runbook covers SSL/TLS certificate management across the homelab, including Let's Encrypt certificates, Cloudflare Origin certificates, and self-signed certificates. It provides procedures for manual renewal, troubleshooting auto-renewal, and emergency certificate fixes.
|
||||
|
||||
## Prerequisites
|
||||
- [ ] SSH access to relevant hosts
|
||||
- [ ] Cloudflare account access (if using Cloudflare)
|
||||
- [ ] Domain DNS control
|
||||
- [ ] Root/sudo privileges on hosts
|
||||
- [ ] Backup of current certificates
|
||||
|
||||
## Metadata
|
||||
- **Estimated Time**: 15-45 minutes
|
||||
- **Risk Level**: Medium (service downtime if misconfigured)
|
||||
- **Requires Downtime**: Minimal (few seconds during reload)
|
||||
- **Reversible**: Yes (can restore old certificates)
|
||||
- **Tested On**: 2026-02-14
|
||||
|
||||
## Certificate Types in Homelab
|
||||
|
||||
| Type | Used For | Renewal Method | Expiration |
|
||||
|------|----------|----------------|------------|
|
||||
| **Let's Encrypt** | Public-facing services | Certbot auto-renewal | 90 days |
|
||||
| **Cloudflare Origin** | Services behind Cloudflare Tunnel | Manual/Cloudflare dashboard | 15 years |
|
||||
| **Synology Certificates** | Synology DSM, services | Synology DSM auto-renewal | 90 days |
|
||||
| **Self-Signed** | Internal/dev services | Manual generation | As configured |
|
||||
|
||||
## Certificate Inventory
|
||||
|
||||
Document your current certificates:
|
||||
|
||||
```bash
|
||||
# Check Let's Encrypt certificates (on Linux hosts)
|
||||
sudo certbot certificates
|
||||
|
||||
# Check Synology certificates
|
||||
# DSM UI → Control Panel → Security → Certificate
|
||||
# Or SSH:
|
||||
sudo cat /usr/syno/etc/certificate/_archive/*/cert.pem | openssl x509 -text -noout
|
||||
|
||||
# Check certificate expiration for any domain
|
||||
echo | openssl s_client -servername service.vish.gg -connect service.vish.gg:443 2>/dev/null | openssl x509 -noout -dates
|
||||
|
||||
# Check all certificates at once
|
||||
for domain in st.vish.gg gf.vish.gg mx.vish.gg; do
|
||||
echo "=== $domain ==="
|
||||
echo | timeout 5 openssl s_client -servername $domain -connect $domain:443 2>/dev/null | openssl x509 -noout -dates
|
||||
echo
|
||||
done
|
||||
```
|
||||
|
||||
Create inventory:
|
||||
```markdown
|
||||
| Domain | Type | Expiry Date | Auto-Renew | Status |
|
||||
|--------|------|-------------|------------|--------|
|
||||
| vish.gg | Let's Encrypt | 2026-05-15 | ✅ Yes | ✅ Valid |
|
||||
| st.vish.gg | Let's Encrypt | 2026-05-15 | ✅ Yes | ✅ Valid |
|
||||
| gf.vish.gg | Let's Encrypt | 2026-05-15 | ✅ Yes | ✅ Valid |
|
||||
```
|
||||
|
||||
## Let's Encrypt Certificate Renewal
|
||||
|
||||
### Automatic Renewal (Certbot)
|
||||
|
||||
Let's Encrypt certificates should auto-renew. Check the renewal setup:
|
||||
|
||||
```bash
|
||||
# Check certbot timer status (systemd)
|
||||
sudo systemctl status certbot.timer
|
||||
|
||||
# Check cron job (if using cron)
|
||||
sudo crontab -l | grep certbot
|
||||
|
||||
# Test renewal (dry-run, doesn't actually renew)
|
||||
sudo certbot renew --dry-run
|
||||
|
||||
# Expected output:
|
||||
# Congratulations, all simulated renewals succeeded
|
||||
```
|
||||
|
||||
### Manual Renewal
|
||||
|
||||
If auto-renewal fails or you need to renew manually:
|
||||
|
||||
```bash
|
||||
# Renew all certificates
|
||||
sudo certbot renew
|
||||
|
||||
# Renew specific certificate
|
||||
sudo certbot renew --cert-name vish.gg
|
||||
|
||||
# Force renewal (even if not expired)
|
||||
sudo certbot renew --force-renewal
|
||||
|
||||
# Renew with verbose output for troubleshooting
|
||||
sudo certbot renew --verbose
|
||||
```
|
||||
|
||||
After renewal, reload web servers:
|
||||
|
||||
```bash
|
||||
# Nginx
|
||||
sudo nginx -t # Test configuration
|
||||
sudo systemctl reload nginx
|
||||
|
||||
# Apache
|
||||
sudo apachectl configtest
|
||||
sudo systemctl reload apache2
|
||||
```
|
||||
|
||||
### Let's Encrypt with Nginx Proxy Manager
|
||||
|
||||
If using Nginx Proxy Manager (NPM):
|
||||
|
||||
1. Open NPM UI (typically port 81)
|
||||
2. Go to **SSL Certificates** tab
|
||||
3. Certificates should auto-renew 30 days before expiry
|
||||
4. To force renewal:
|
||||
- Click the certificate
|
||||
- Click **Renew** button
|
||||
5. No service reload needed (NPM handles it)
|
||||
|
||||
## Synology Certificate Renewal
|
||||
|
||||
### Automatic Renewal on Synology NAS
|
||||
|
||||
```bash
|
||||
# SSH to Synology NAS (Atlantis or Calypso)
|
||||
ssh atlantis # or calypso
|
||||
|
||||
# Check certificate status
|
||||
sudo /usr/syno/sbin/syno-letsencrypt list
|
||||
|
||||
# Force renewal check
|
||||
sudo /usr/syno/sbin/syno-letsencrypt renew-all
|
||||
|
||||
# Check renewal logs
|
||||
sudo cat /var/log/letsencrypt/letsencrypt.log
|
||||
|
||||
# Verify certificate expiry
|
||||
sudo openssl x509 -in /usr/syno/etc/certificate/system/default/cert.pem -text -noout | grep "Not After"
|
||||
```
|
||||
|
||||
### Via Synology DSM UI
|
||||
|
||||
1. Log in to DSM
|
||||
2. **Control Panel** → **Security** → **Certificate**
|
||||
3. Select certificate → Click **Renew**
|
||||
4. DSM will automatically renew and apply
|
||||
5. No manual reload needed
|
||||
|
||||
### Synology Certificate Configuration
|
||||
|
||||
Enable auto-renewal in DSM:
|
||||
1. **Control Panel** → **Security** → **Certificate**
|
||||
2. Click **Settings** button
|
||||
3. Check **Auto-renew certificate**
|
||||
4. Synology will renew 30 days before expiry
|
||||
|
||||
## Stoatchat Certificates (Gaming VPS)
|
||||
|
||||
The Stoatchat gaming server uses Let's Encrypt with Certbot:
|
||||
|
||||
```bash
|
||||
# SSH to gaming VPS
|
||||
ssh root@gaming-vps
|
||||
|
||||
# Check certificates
|
||||
sudo certbot certificates
|
||||
|
||||
# Domains covered:
|
||||
# - st.vish.gg
|
||||
# - api.st.vish.gg
|
||||
# - events.st.vish.gg
|
||||
# - files.st.vish.gg
|
||||
# - proxy.st.vish.gg
|
||||
# - voice.st.vish.gg
|
||||
|
||||
# Renew all
|
||||
sudo certbot renew
|
||||
|
||||
# Reload Nginx
|
||||
sudo systemctl reload nginx
|
||||
```
|
||||
|
||||
Auto-renewal cron:
|
||||
```bash
|
||||
# Check certbot timer
|
||||
sudo systemctl status certbot.timer
|
||||
|
||||
# Or check cron
|
||||
sudo crontab -l | grep certbot
|
||||
```
|
||||
|
||||
## Cloudflare Origin Certificates
|
||||
|
||||
For services using Cloudflare Tunnel:
|
||||
|
||||
### Generate New Origin Certificate
|
||||
|
||||
1. Log in to Cloudflare Dashboard
|
||||
2. Select domain (vish.gg)
|
||||
3. **SSL/TLS** → **Origin Server**
|
||||
4. Click **Create Certificate**
|
||||
5. Configure:
|
||||
- **Private key type**: RSA (2048)
|
||||
- **Hostnames**: *.vish.gg, vish.gg
|
||||
- **Certificate validity**: 15 years
|
||||
6. Copy certificate and private key
|
||||
7. Save to secure location
|
||||
|
||||
### Install Origin Certificate
|
||||
|
||||
```bash
|
||||
# SSH to target host
|
||||
ssh [host]
|
||||
|
||||
# Create certificate files
|
||||
sudo nano /etc/ssl/cloudflare/cert.pem
|
||||
# Paste certificate
|
||||
|
||||
sudo nano /etc/ssl/cloudflare/key.pem
|
||||
# Paste private key
|
||||
|
||||
# Set permissions
|
||||
sudo chmod 644 /etc/ssl/cloudflare/cert.pem
|
||||
sudo chmod 600 /etc/ssl/cloudflare/key.pem
|
||||
|
||||
# Update Nginx configuration
|
||||
sudo nano /etc/nginx/sites-available/[service]
|
||||
|
||||
# Use new certificate
|
||||
ssl_certificate /etc/ssl/cloudflare/cert.pem;
|
||||
ssl_certificate_key /etc/ssl/cloudflare/key.pem;
|
||||
|
||||
# Test and reload
|
||||
sudo nginx -t
|
||||
sudo systemctl reload nginx
|
||||
```
|
||||
|
||||
## Self-Signed Certificates (Internal/Dev)
|
||||
|
||||
For internal-only services not exposed publicly:
|
||||
|
||||
### Generate Self-Signed Certificate
|
||||
|
||||
```bash
|
||||
# Generate 10-year self-signed certificate
|
||||
sudo openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
|
||||
-keyout /etc/ssl/private/selfsigned.key \
|
||||
-out /etc/ssl/certs/selfsigned.crt \
|
||||
-subj "/C=US/ST=State/L=City/O=Homelab/CN=internal.vish.local"
|
||||
|
||||
# Generate with SAN (Subject Alternative Names) for multiple domains
|
||||
sudo openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
|
||||
-keyout /etc/ssl/private/selfsigned.key \
|
||||
-out /etc/ssl/certs/selfsigned.crt \
|
||||
-subj "/C=US/ST=State/L=City/O=Homelab/CN=*.vish.local" \
|
||||
-addext "subjectAltName=DNS:*.vish.local,DNS:vish.local"
|
||||
|
||||
# Set permissions
|
||||
sudo chmod 600 /etc/ssl/private/selfsigned.key
|
||||
sudo chmod 644 /etc/ssl/certs/selfsigned.crt
|
||||
```
|
||||
|
||||
### Install in Services
|
||||
|
||||
Update Docker Compose to mount certificates:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
service:
|
||||
volumes:
|
||||
- /etc/ssl/certs/selfsigned.crt:/etc/ssl/certs/cert.pem:ro
|
||||
- /etc/ssl/private/selfsigned.key:/etc/ssl/private/key.pem:ro
|
||||
```
|
||||
|
||||
## Monitoring Certificate Expiration
|
||||
|
||||
### Set Up Expiration Alerts
|
||||
|
||||
Create a certificate monitoring script:
|
||||
|
||||
```bash
|
||||
sudo nano /usr/local/bin/check-certificates.sh
|
||||
```
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Certificate Expiration Monitoring Script
|
||||
|
||||
DOMAINS=(
|
||||
"vish.gg"
|
||||
"st.vish.gg"
|
||||
"gf.vish.gg"
|
||||
"mx.vish.gg"
|
||||
)
|
||||
|
||||
ALERT_DAYS=30 # Alert if expiring within 30 days
|
||||
WEBHOOK_URL="https://ntfy.sh/REDACTED_TOPIC" # Your notification webhook
|
||||
|
||||
for domain in "${DOMAINS[@]}"; do
|
||||
echo "Checking $domain..."
|
||||
|
||||
# Get certificate expiration date
|
||||
expiry=$(echo | openssl s_client -servername $domain -connect $domain:443 2>/dev/null | \
|
||||
openssl x509 -noout -dates | grep "notAfter" | cut -d= -f2)
|
||||
|
||||
# Convert to epoch time
|
||||
expiry_epoch=$(date -d "$expiry" +%s)
|
||||
current_epoch=$(date +%s)
|
||||
days_left=$(( ($expiry_epoch - $current_epoch) / 86400 ))
|
||||
|
||||
echo "$domain expires in $days_left days"
|
||||
|
||||
if [ $days_left -lt $ALERT_DAYS ]; then
|
||||
# Send alert
|
||||
curl -H "Title: Certificate Expiring Soon" \
|
||||
-H "Priority: high" \
|
||||
-H "Tags: warning,certificate" \
|
||||
-d "Certificate for $domain expires in $days_left days!" \
|
||||
$WEBHOOK_URL
|
||||
|
||||
echo "⚠️ Alert sent for $domain"
|
||||
fi
|
||||
echo
|
||||
done
|
||||
```
|
||||
|
||||
Make executable and add to cron:
|
||||
```bash
|
||||
sudo chmod +x /usr/local/bin/check-certificates.sh
|
||||
|
||||
# Add to cron (daily at 9 AM)
|
||||
(crontab -l 2>/dev/null; echo "0 9 * * * /usr/local/bin/check-certificates.sh") | crontab -
|
||||
```
|
||||
|
||||
### Grafana Dashboard
|
||||
|
||||
Add certificate monitoring to Grafana:
|
||||
|
||||
```bash
|
||||
# Install blackbox_exporter for HTTPS probing
|
||||
# Add to prometheus.yml:
|
||||
|
||||
scrape_configs:
|
||||
- job_name: 'blackbox'
|
||||
metrics_path: /probe
|
||||
params:
|
||||
module: [http_2xx]
|
||||
static_configs:
|
||||
- targets:
|
||||
- https://vish.gg
|
||||
- https://st.vish.gg
|
||||
- https://gf.vish.gg
|
||||
relabel_configs:
|
||||
- source_labels: [__address__]
|
||||
target_label: __param_target
|
||||
- source_labels: [__param_target]
|
||||
target_label: instance
|
||||
- target_label: __address__
|
||||
replacement: blackbox-exporter:9115
|
||||
|
||||
# Create alert rule:
|
||||
- alert: SSLCertificateExpiring
|
||||
expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "SSL certificate expiring soon"
|
||||
description: "SSL certificate for {{ $labels.instance }} expires in {{ $value | REDACTED_APP_PASSWORD }}"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Certbot Renewal Failing
|
||||
|
||||
**Symptoms**: `certbot renew` fails with DNS or HTTP challenge errors
|
||||
|
||||
**Solutions**:
|
||||
|
||||
```bash
|
||||
# Check detailed error logs
|
||||
sudo certbot renew --verbose
|
||||
|
||||
# Common issues:
|
||||
|
||||
# 1. Port 80/443 not accessible
|
||||
sudo ufw status # Check firewall
|
||||
sudo netstat -tlnp | grep :80 # Check if port is listening
|
||||
|
||||
# 2. DNS not resolving correctly
|
||||
dig vish.gg # Verify DNS points to correct IP
|
||||
|
||||
# 3. Rate limits hit
|
||||
# Let's Encrypt has rate limits: 50 certificates per domain per week
|
||||
# Wait 7 days or use --staging for testing
|
||||
|
||||
# 4. Webroot path incorrect
|
||||
sudo certbot renew --webroot -w /var/www/html
|
||||
|
||||
# 5. Try force renewal with different challenge
|
||||
sudo certbot renew --force-renewal --preferred-challenges dns
|
||||
```
|
||||
|
||||
### Issue: Certificate Valid But Browser Shows Warning
|
||||
|
||||
**Symptoms**: Certificate is valid but browsers show security warning
|
||||
|
||||
**Solutions**:
|
||||
|
||||
```bash
|
||||
# Check certificate chain
|
||||
openssl s_client -connect vish.gg:443 -showcerts
|
||||
|
||||
# Ensure intermediate certificates are included
|
||||
# Nginx: Use fullchain.pem, not cert.pem
|
||||
ssl_certificate /etc/letsencrypt/live/vish.gg/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/vish.gg/privkey.pem;
|
||||
|
||||
# Test SSL configuration
|
||||
curl -I https://vish.gg
|
||||
# Or use: https://www.ssllabs.com/ssltest/
|
||||
```
|
||||
|
||||
### Issue: Synology Certificate Not Auto-Renewing
|
||||
|
||||
**Symptoms**: DSM certificate expired or shows renewal error
|
||||
|
||||
**Solutions**:
|
||||
|
||||
```bash
|
||||
# SSH to Synology
|
||||
ssh atlantis
|
||||
|
||||
# Check renewal logs
|
||||
sudo cat /var/log/letsencrypt/letsencrypt.log
|
||||
|
||||
# Common issues:
|
||||
|
||||
# 1. Port 80 forwarding
|
||||
# Ensure port 80 is forwarded to NAS during renewal
|
||||
|
||||
# 2. Domain validation
|
||||
# Check DNS points to correct external IP
|
||||
|
||||
# 3. Force renewal
|
||||
sudo /usr/syno/sbin/syno-letsencrypt renew-all
|
||||
|
||||
# 4. Restart certificate service
|
||||
sudo synosystemctl restart nginx
|
||||
```
|
||||
|
||||
### Issue: Nginx Won't Reload After Certificate Update
|
||||
|
||||
**Symptoms**: `nginx -t` shows SSL errors
|
||||
|
||||
**Solutions**:
|
||||
|
||||
```bash
|
||||
# Test Nginx configuration
|
||||
sudo nginx -t
|
||||
|
||||
# Common errors:
|
||||
|
||||
# 1. Certificate path incorrect
|
||||
# Fix: Update nginx config with correct path
|
||||
|
||||
# 2. Certificate and key mismatch
|
||||
# Verify:
|
||||
sudo openssl x509 -noout -modulus -in cert.pem | openssl md5
|
||||
sudo openssl rsa -noout -modulus -in key.pem | openssl md5
|
||||
# MD5 sums should match
|
||||
|
||||
# 3. Permission issues
|
||||
sudo chmod 644 /etc/ssl/certs/cert.pem
|
||||
sudo chmod 600 /etc/ssl/private/key.pem
|
||||
sudo chown root:root /etc/ssl/certs/cert.pem /etc/ssl/private/key.pem
|
||||
|
||||
# 4. SELinux blocking (if enabled)
|
||||
sudo setsebool -P httpd_read_user_content 1
|
||||
```
|
||||
|
||||
## Emergency Certificate Fix
|
||||
|
||||
If a certificate expires and services are down:
|
||||
|
||||
### Quick Fix: Use Self-Signed Temporarily
|
||||
|
||||
```bash
|
||||
# Generate emergency self-signed certificate
|
||||
sudo openssl req -x509 -nodes -days 30 -newkey rsa:2048 \
|
||||
-keyout /tmp/emergency.key \
|
||||
-out /tmp/emergency.crt \
|
||||
-subj "/CN=*.vish.gg"
|
||||
|
||||
# Update Nginx to use emergency cert
|
||||
sudo nano /etc/nginx/sites-available/default
|
||||
|
||||
ssl_certificate /tmp/emergency.crt;
|
||||
ssl_certificate_key /tmp/emergency.key;
|
||||
|
||||
# Reload Nginx
|
||||
sudo nginx -t && sudo systemctl reload nginx
|
||||
|
||||
# Services are now accessible (with browser warning)
|
||||
# Then fix proper certificate renewal
|
||||
```
|
||||
|
||||
### Restore from Backup
|
||||
|
||||
```bash
|
||||
# If certificates were backed up
|
||||
sudo cp /backup/letsencrypt/archive/vish.gg/* /etc/letsencrypt/archive/vish.gg/
|
||||
|
||||
# Update symlinks
|
||||
sudo certbot certificates # Shows current status
|
||||
sudo certbot install --cert-name vish.gg
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Renewal Schedule
|
||||
- Let's Encrypt certificates renew at 60 days (30 days before expiry)
|
||||
- Check certificates monthly
|
||||
- Set up expiration alerts
|
||||
- Test renewal process quarterly
|
||||
|
||||
### Backup Certificates
|
||||
```bash
|
||||
# Backup Let's Encrypt certificates
|
||||
sudo tar czf ~/letsencrypt-backup-$(date +%Y%m%d).tar.gz /etc/letsencrypt/
|
||||
|
||||
# Backup Synology certificates
|
||||
# Done via Synology backup tasks
|
||||
|
||||
# Store backups securely (encrypted, off-site)
|
||||
```
|
||||
|
||||
### Documentation
|
||||
- Document which certificates are used where
|
||||
- Keep inventory of expiration dates
|
||||
- Document renewal procedures
|
||||
- Note any special configurations
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
After certificate renewal:
|
||||
|
||||
- [ ] Certificate renewed successfully
|
||||
- [ ] Certificate expiry date extended
|
||||
- [ ] Web servers reloaded without errors
|
||||
- [ ] All services accessible via HTTPS
|
||||
- [ ] No browser security warnings
|
||||
- [ ] Certificate chain complete
|
||||
- [ ] Auto-renewal still enabled
|
||||
- [ ] Monitoring updated (if needed)
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Infrastructure Overview](../infrastructure/INFRASTRUCTURE_OVERVIEW.md)
|
||||
- [Nginx Configuration](../infrastructure/networking.md)
|
||||
- [Cloudflare Tunnels Setup](../infrastructure/cloudflare-tunnels-setup.md)
|
||||
- [Emergency Access Guide](../troubleshooting/EMERGENCY_ACCESS_GUIDE.md)
|
||||
|
||||
## Change Log
|
||||
|
||||
- 2026-02-14 - Initial creation
|
||||
- 2026-02-14 - Added monitoring and troubleshooting sections
|
||||
Reference in New Issue
Block a user