homelab-optimized/docs/admin/testing-procedures.md

Testing Procedures

Testing guidelines for the homelab infrastructure

---

Overview

This document outlines testing procedures for deploying new services, making infrastructure changes, and validating functionality.

---

Pre-Deployment Testing

New Service Checklist

• Review Docker image (official, stars, updates)
• Check for security vulnerabilities
• Verify resource requirements
• Test locally first
• Verify compose syntax
• Check port availability
• Test volume paths

Compose Validation

# Validate syntax
docker-compose config --quiet

# Check for errors
docker-compose up --dry-run

# Pull images
docker-compose pull

---

Local Testing

Docker Desktop / Mini Setup

1. Create test compose file
2. Run on local machine
3. Verify all features work
4. Document any issues

Test Environment

If available, use staging:

• Staging host: seattle VM
• Test domain: *.test.vish.local
• Shared internally only

---

Integration Testing

Authentik SSO

# Test login flow
1. Open service
2. Click "Login with Authentik"
3. Verify redirect to Authentik
4. Enter credentials
5. Verify return to service
6. Check user profile

Nginx Proxy Manager

# Test proxy host
curl -H "Host: service.vish.local" http://localhost

# Test SSL
curl -k https://service.vish.gg

# Check headers
curl -I https://service.vish.gg

Database Connections

# PostgreSQL
docker exec <container> psql -U user -c "SELECT 1"

# Test from application
docker exec <app> nc -zv db 5432

---

Monitoring Validation

Prometheus Targets

1. Open Prometheus UI
2. Go to Status → Targets
3. Verify all targets are UP
4. Check for scrape errors

Alert Testing

# Trigger test alert
curl -X POST http://alertmanager:9093/api/v1/alerts \
  -H "Content-Type: application/json" \
  -d '[{
    "labels": {
      "alertname": "TestAlert",
      "severity": "critical"
    },
    "annotations": {
      "summary": "Test alert"
    }
  }]'

Grafana Dashboards

• All panels load
• Data populates
• No errors in console
• Alerts configured

---

Backup Testing

Full Backup Test

# Run backup
ansible-playbook ansible/automation/playbooks/backup_configs.yml
ansible-playbook ansible/automation/playbooks/backup_databases.yml

# Verify backup files exist
ls -la /backup/

# Test restore to test environment
# (do NOT overwrite production!)

Restore Procedure Test

1. Stop service
2. Restore data from backup
3. Start service
4. Verify functionality
5. Check logs for errors

---

Performance Testing

Load Testing

# Using hey or ab
hey -n 1000 -c 10 https://service.vish.gg

# Check response times
curl -w "@curl-format.txt" -o /dev/null -s https://service.vish.gg

# curl-format.txt:
# time_namelookup:  %{time_namelookup}\n
# time_connect:     %{time_connect}\n
# time_appconnect:  %{time_appconnect}\n
# time_redirect:    %{time_redirect}\n
# time_pretransfer: %{time_pretransfer}\n
# time_starttransfer: %{time_starttransfer}\n
# time_total:       %{time_total}\n

Resource Testing

# Monitor during load
docker stats --no-stream

# Check for OOM kills
dmesg | grep -i "out of memory"

# Monitor disk I/O
iostat -x 1

---

Security Testing

Vulnerability Scanning

# Trivy scan
trivy image --severity HIGH,CRITICAL <image>

# Check for secrets
trivy fs --security-checks secrets /path/to/compose

# Docker scan
docker scan <image>

SSL/TLS Testing

# SSL Labs
# Visit: https://www.ssllabs.com/ssltest/

# CLI check
openssl s_client -connect service.vish.gg:443

# Check certificates
certinfo service.vish.gg

---

Network Testing

Connectivity

# Port scan
nmap -p 1-1000 192.168.0.x

# DNS check
dig service.vish.local
nslookup service.vish.local

# traceroute
traceroute service.vish.gg

Firewall Testing

# Check open ports
ss -tulpn

# Test from outside
# Use online port scanner

# Test blocked access
curl -I http://internal-service:port
# Should fail without VPN

---

Regression Testing

After Updates

1. Check service starts
2. Verify all features
3. Test SSO if enabled
4. Check monitoring
5. Verify backups

Critical Path Tests

Path	Steps
External access	VPN → NPM → Service
SSO login	Service → Auth → Dashboard
Media playback	Request → Download → Play
Backup restore	Stop → Restore → Verify → Start

---

Acceptance Criteria

New Service

• Starts without errors
• UI accessible
• Basic function works
• SSO configured (if supported)
• Monitoring enabled
• Backup configured
• Documentation created

Infrastructure Change

• All services running
• No new alerts
• Monitoring healthy
• Backups completed
• Users notified (if needed)

---

Links

• Monitoring Architecture
• Backup Procedures
• Disaster Recovery