homelab-optimized/docs/runbooks/certificate-renewal.md

SSL/TLS Certificate Renewal Runbook

Overview

This runbook covers SSL/TLS certificate management across the homelab, including Let's Encrypt certificates, Cloudflare Origin certificates, and self-signed certificates. It provides procedures for manual renewal, troubleshooting auto-renewal, and emergency certificate fixes.

Prerequisites

• SSH access to relevant hosts
• Cloudflare account access (if using Cloudflare)
• Domain DNS control
• Root/sudo privileges on hosts
• Backup of current certificates

Metadata

• Estimated Time: 15-45 minutes
• Risk Level: Medium (service downtime if misconfigured)
• Requires Downtime: Minimal (few seconds during reload)
• Reversible: Yes (can restore old certificates)
• Tested On: 2026-02-14

Certificate Types in Homelab

Type	Used For	Renewal Method	Expiration
Let's Encrypt	Public-facing services	Certbot auto-renewal	90 days
Cloudflare Origin	Services behind Cloudflare Tunnel	Manual/Cloudflare dashboard	15 years
Synology Certificates	Synology DSM, services	Synology DSM auto-renewal	90 days
Self-Signed	Internal/dev services	Manual generation	As configured

Certificate Inventory

Document your current certificates:

# Check Let's Encrypt certificates (on Linux hosts)
sudo certbot certificates

# Check Synology certificates
# DSM UI → Control Panel → Security → Certificate
# Or SSH:
sudo cat /usr/syno/etc/certificate/_archive/*/cert.pem | openssl x509 -text -noout

# Check certificate expiration for any domain
echo | openssl s_client -servername service.vish.gg -connect service.vish.gg:443 2>/dev/null | openssl x509 -noout -dates

# Check all certificates at once
for domain in st.vish.gg gf.vish.gg mx.vish.gg; do
    echo "=== $domain ==="
    echo | timeout 5 openssl s_client -servername $domain -connect $domain:443 2>/dev/null | openssl x509 -noout -dates
    echo
done

Create inventory:

| Domain | Type | Expiry Date | Auto-Renew | Status |
|--------|------|-------------|------------|--------|
| vish.gg | Let's Encrypt | 2026-05-15 | ✅ Yes | ✅ Valid |
| st.vish.gg | Let's Encrypt | 2026-05-15 | ✅ Yes | ✅ Valid |
| gf.vish.gg | Let's Encrypt | 2026-05-15 | ✅ Yes | ✅ Valid |

Let's Encrypt Certificate Renewal

Automatic Renewal (Certbot)

Let's Encrypt certificates should auto-renew. Check the renewal setup:

# Check certbot timer status (systemd)
sudo systemctl status certbot.timer

# Check cron job (if using cron)
sudo crontab -l | grep certbot

# Test renewal (dry-run, doesn't actually renew)
sudo certbot renew --dry-run

# Expected output:
# Congratulations, all simulated renewals succeeded

Manual Renewal

If auto-renewal fails or you need to renew manually:

# Renew all certificates
sudo certbot renew

# Renew specific certificate
sudo certbot renew --cert-name vish.gg

# Force renewal (even if not expired)
sudo certbot renew --force-renewal

# Renew with verbose output for troubleshooting
sudo certbot renew --verbose

After renewal, reload web servers:

# Nginx
sudo nginx -t  # Test configuration
sudo systemctl reload nginx

# Apache
sudo apachectl configtest
sudo systemctl reload apache2

Let's Encrypt with Nginx Proxy Manager

If using Nginx Proxy Manager (NPM):

1. Open NPM UI (typically port 81)
2. Go to SSL Certificates tab
3. Certificates should auto-renew 30 days before expiry
4. To force renewal:
   • Click the certificate
   • Click Renew button
5. No service reload needed (NPM handles it)

Synology Certificate Renewal

Automatic Renewal on Synology NAS

# SSH to Synology NAS (Atlantis or Calypso)
ssh atlantis  # or calypso

# Check certificate status
sudo /usr/syno/sbin/syno-letsencrypt list

# Force renewal check
sudo /usr/syno/sbin/syno-letsencrypt renew-all

# Check renewal logs
sudo cat /var/log/letsencrypt/letsencrypt.log

# Verify certificate expiry
sudo openssl x509 -in /usr/syno/etc/certificate/system/default/cert.pem -text -noout | grep "Not After"

Via Synology DSM UI

1. Log in to DSM
2. Control Panel → Security → Certificate
3. Select certificate → Click Renew
4. DSM will automatically renew and apply
5. No manual reload needed

Synology Certificate Configuration

Enable auto-renewal in DSM:

1. Control Panel → Security → Certificate
2. Click Settings button
3. Check Auto-renew certificate
4. Synology will renew 30 days before expiry

Stoatchat Certificates (Gaming VPS)

The Stoatchat gaming server uses Let's Encrypt with Certbot:

# SSH to gaming VPS
ssh root@gaming-vps

# Check certificates
sudo certbot certificates

# Domains covered:
# - st.vish.gg
# - api.st.vish.gg
# - events.st.vish.gg
# - files.st.vish.gg
# - proxy.st.vish.gg
# - voice.st.vish.gg

# Renew all
sudo certbot renew

# Reload Nginx
sudo systemctl reload nginx

Auto-renewal cron:

# Check certbot timer
sudo systemctl status certbot.timer

# Or check cron
sudo crontab -l | grep certbot

Cloudflare Origin Certificates

For services using Cloudflare Tunnel:

Generate New Origin Certificate

1. Log in to Cloudflare Dashboard
2. Select domain (vish.gg)
3. SSL/TLS → Origin Server
4. Click Create Certificate
5. Configure:
   • Private key type: RSA (2048)
   • Hostnames: *.vish.gg, vish.gg
   • Certificate validity: 15 years
6. Copy certificate and private key
7. Save to secure location

Install Origin Certificate

# SSH to target host
ssh [host]

# Create certificate files
sudo nano /etc/ssl/cloudflare/cert.pem
# Paste certificate

sudo nano /etc/ssl/cloudflare/key.pem
# Paste private key

# Set permissions
sudo chmod 644 /etc/ssl/cloudflare/cert.pem
sudo chmod 600 /etc/ssl/cloudflare/key.pem

# Update Nginx configuration
sudo nano /etc/nginx/sites-available/[service]

# Use new certificate
ssl_certificate /etc/ssl/cloudflare/cert.pem;
ssl_certificate_key /etc/ssl/cloudflare/key.pem;

# Test and reload
sudo nginx -t
sudo systemctl reload nginx

Self-Signed Certificates (Internal/Dev)

For internal-only services not exposed publicly:

Generate Self-Signed Certificate

# Generate 10-year self-signed certificate
sudo openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
    -keyout /etc/ssl/private/selfsigned.key \
    -out /etc/ssl/certs/selfsigned.crt \
    -subj "/C=US/ST=State/L=City/O=Homelab/CN=internal.vish.local"

# Generate with SAN (Subject Alternative Names) for multiple domains
sudo openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
    -keyout /etc/ssl/private/selfsigned.key \
    -out /etc/ssl/certs/selfsigned.crt \
    -subj "/C=US/ST=State/L=City/O=Homelab/CN=*.vish.local" \
    -addext "subjectAltName=DNS:*.vish.local,DNS:vish.local"

# Set permissions
sudo chmod 600 /etc/ssl/private/selfsigned.key
sudo chmod 644 /etc/ssl/certs/selfsigned.crt

Install in Services

Update Docker Compose to mount certificates:

services:
  service:
    volumes:
      - /etc/ssl/certs/selfsigned.crt:/etc/ssl/certs/cert.pem:ro
      - /etc/ssl/private/selfsigned.key:/etc/ssl/private/key.pem:ro

Monitoring Certificate Expiration

Set Up Expiration Alerts

Create a certificate monitoring script:

sudo nano /usr/local/bin/check-certificates.sh

#!/bin/bash
# Certificate Expiration Monitoring Script

DOMAINS=(
    "vish.gg"
    "st.vish.gg"
    "gf.vish.gg"
    "mx.vish.gg"
)

ALERT_DAYS=30  # Alert if expiring within 30 days
WEBHOOK_URL="https://ntfy.sh/REDACTED_TOPIC"  # Your notification webhook

for domain in "${DOMAINS[@]}"; do
    echo "Checking $domain..."

    # Get certificate expiration date
    expiry=$(echo | openssl s_client -servername $domain -connect $domain:443 2>/dev/null | \
             openssl x509 -noout -dates | grep "notAfter" | cut -d= -f2)

    # Convert to epoch time
    expiry_epoch=$(date -d "$expiry" +%s)
    current_epoch=$(date +%s)
    days_left=$(( ($expiry_epoch - $current_epoch) / 86400 ))

    echo "$domain expires in $days_left days"

    if [ $days_left -lt $ALERT_DAYS ]; then
        # Send alert
        curl -H "Title: Certificate Expiring Soon" \
             -H "Priority: high" \
             -H "Tags: warning,certificate" \
             -d "Certificate for $domain expires in $days_left days!" \
             $WEBHOOK_URL

        echo "⚠️  Alert sent for $domain"
    fi
    echo
done

Make executable and add to cron:

sudo chmod +x /usr/local/bin/check-certificates.sh

# Add to cron (daily at 9 AM)
(crontab -l 2>/dev/null; echo "0 9 * * * /usr/local/bin/check-certificates.sh") | crontab -

Grafana Dashboard

Add certificate monitoring to Grafana:

# Install blackbox_exporter for HTTPS probing
# Add to prometheus.yml:

scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://vish.gg
        - https://st.vish.gg
        - https://gf.vish.gg
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

# Create alert rule:
- alert: SSLCertificateExpiring
  expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
  labels:
    severity: warning
  annotations:
    summary: "SSL certificate expiring soon"
    description: "SSL certificate for {{ $labels.instance }} expires in {{ $value | REDACTED_APP_PASSWORD }}"

Troubleshooting

Issue: Certbot Renewal Failing

Symptoms: certbot renew fails with DNS or HTTP challenge errors

Solutions:

# Check detailed error logs
sudo certbot renew --verbose

# Common issues:

# 1. Port 80/443 not accessible
sudo ufw status  # Check firewall
sudo netstat -tlnp | grep :80  # Check if port is listening

# 2. DNS not resolving correctly
dig vish.gg  # Verify DNS points to correct IP

# 3. Rate limits hit
# Let's Encrypt has rate limits: 50 certificates per domain per week
# Wait 7 days or use --staging for testing

# 4. Webroot path incorrect
sudo certbot renew --webroot -w /var/www/html

# 5. Try force renewal with different challenge
sudo certbot renew --force-renewal --preferred-challenges dns

Issue: Certificate Valid But Browser Shows Warning

Symptoms: Certificate is valid but browsers show security warning

Solutions:

# Check certificate chain
openssl s_client -connect vish.gg:443 -showcerts

# Ensure intermediate certificates are included
# Nginx: Use fullchain.pem, not cert.pem
ssl_certificate /etc/letsencrypt/live/vish.gg/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/vish.gg/privkey.pem;

# Test SSL configuration
curl -I https://vish.gg
# Or use: https://www.ssllabs.com/ssltest/

Issue: Synology Certificate Not Auto-Renewing

Symptoms: DSM certificate expired or shows renewal error

Solutions:

# SSH to Synology
ssh atlantis

# Check renewal logs
sudo cat /var/log/letsencrypt/letsencrypt.log

# Common issues:

# 1. Port 80 forwarding
# Ensure port 80 is forwarded to NAS during renewal

# 2. Domain validation
# Check DNS points to correct external IP

# 3. Force renewal
sudo /usr/syno/sbin/syno-letsencrypt renew-all

# 4. Restart certificate service
sudo synosystemctl restart nginx

Issue: Nginx Won't Reload After Certificate Update

Symptoms: nginx -t shows SSL errors

Solutions:

# Test Nginx configuration
sudo nginx -t

# Common errors:

# 1. Certificate path incorrect
# Fix: Update nginx config with correct path

# 2. Certificate and key mismatch
# Verify:
sudo openssl x509 -noout -modulus -in cert.pem | openssl md5
sudo openssl rsa -noout -modulus -in key.pem | openssl md5
# MD5 sums should match

# 3. Permission issues
sudo chmod 644 /etc/ssl/certs/cert.pem
sudo chmod 600 /etc/ssl/private/key.pem
sudo chown root:root /etc/ssl/certs/cert.pem /etc/ssl/private/key.pem

# 4. SELinux blocking (if enabled)
sudo setsebool -P httpd_read_user_content 1

Emergency Certificate Fix

If a certificate expires and services are down:

Quick Fix: Use Self-Signed Temporarily

# Generate emergency self-signed certificate
sudo openssl req -x509 -nodes -days 30 -newkey rsa:2048 \
    -keyout /tmp/emergency.key \
    -out /tmp/emergency.crt \
    -subj "/CN=*.vish.gg"

# Update Nginx to use emergency cert
sudo nano /etc/nginx/sites-available/default

ssl_certificate /tmp/emergency.crt;
ssl_certificate_key /tmp/emergency.key;

# Reload Nginx
sudo nginx -t && sudo systemctl reload nginx

# Services are now accessible (with browser warning)
# Then fix proper certificate renewal

Restore from Backup

# If certificates were backed up
sudo cp /backup/letsencrypt/archive/vish.gg/* /etc/letsencrypt/archive/vish.gg/

# Update symlinks
sudo certbot certificates  # Shows current status
sudo certbot install --cert-name vish.gg

Best Practices

Renewal Schedule

• Let's Encrypt certificates renew at 60 days (30 days before expiry)
• Check certificates monthly
• Set up expiration alerts
• Test renewal process quarterly

Backup Certificates

# Backup Let's Encrypt certificates
sudo tar czf ~/letsencrypt-backup-$(date +%Y%m%d).tar.gz /etc/letsencrypt/

# Backup Synology certificates
# Done via Synology backup tasks

# Store backups securely (encrypted, off-site)

Documentation

• Document which certificates are used where
• Keep inventory of expiration dates
• Document renewal procedures
• Note any special configurations

Verification Checklist

After certificate renewal:

• Certificate renewed successfully
• Certificate expiry date extended
• Web servers reloaded without errors
• All services accessible via HTTPS
• No browser security warnings
• Certificate chain complete
• Auto-renewal still enabled
• Monitoring updated (if needed)

Related Documentation

• Infrastructure Overview
• Nginx Configuration
• Cloudflare Tunnels Setup
• Emergency Access Guide

Change Log

• 2026-02-14 - Initial creation
• 2026-02-14 - Added monitoring and troubleshooting sections