Sanitized mirror from private repository - 2026-04-05 12:40:35 UTC

docs/admin/testing-procedures.md

@@ -0,0 +1,303 @@
+# Testing Procedures
+
+*Testing guidelines for the homelab infrastructure*
+
+---
+
+## Overview
+
+This document outlines testing procedures for deploying new services, making infrastructure changes, and validating functionality.
+
+---
+
+## Pre-Deployment Testing
+
+### New Service Checklist
+
+- [ ] Review Docker image (official, stars, updates)
+- [ ] Check for security vulnerabilities
+- [ ] Verify resource requirements
+- [ ] Test locally first
+- [ ] Verify compose syntax
+- [ ] Check port availability
+- [ ] Test volume paths
+
+### Compose Validation
+
+```bash
+# Validate syntax
+docker-compose config --quiet
+
+# Check for errors
+docker-compose up --dry-run
+
+# Pull images
+docker-compose pull
+```
+
+---
+
+## Local Testing
+
+### Docker Desktop / Mini Setup
+
+1. Create test compose file
+2. Run on local machine
+3. Verify all features work
+4. Document any issues
+
+### Test Environment
+
+If available, use staging:
+- Staging host: `seattle` VM
+- Test domain: `*.test.vish.local`
+- Shared internally only
+
+---
+
+## Integration Testing
+
+### Authentik SSO
+
+```bash
+# Test login flow
+1. Open service
+2. Click "Login with Authentik"
+3. Verify redirect to Authentik
+4. Enter credentials
+5. Verify return to service
+6. Check user profile
+```
+
+### Nginx Proxy Manager
+
+```bash
+# Test proxy host
+curl -H "Host: service.vish.local" http://localhost
+
+# Test SSL
+curl -k https://service.vish.gg
+
+# Check headers
+curl -I https://service.vish.gg
+```
+
+### Database Connections
+
+```bash
+# PostgreSQL
+docker exec <container> psql -U user -c "SELECT 1"
+
+# Test from application
+docker exec <app> nc -zv db 5432
+```
+
+---
+
+## Monitoring Validation
+
+### Prometheus Targets
+
+1. Open Prometheus UI
+2. Go to Status → Targets
+3. Verify all targets are UP
+4. Check for scrape errors
+
+### Alert Testing
+
+```bash
+# Trigger test alert
+curl -X POST http://alertmanager:9093/api/v1/alerts \
+  -H "Content-Type: application/json" \
+  -d '[{
+    "labels": {
+      "alertname": "TestAlert",
+      "severity": "critical"
+    },
+    "annotations": {
+      "summary": "Test alert"
+    }
+  }]'
+```
+
+### Grafana Dashboards
+
+- [ ] All panels load
+- [ ] Data populates
+- [ ] No errors in console
+- [ ] Alerts configured
+
+---
+
+## Backup Testing
+
+### Full Backup Test
+
+```bash
+# Run backup
+ansible-playbook ansible/automation/playbooks/backup_configs.yml
+ansible-playbook ansible/automation/playbooks/backup_databases.yml
+
+# Verify backup files exist
+ls -la /backup/
+
+# Test restore to test environment
+# (do NOT overwrite production!)
+```
+
+### Restore Procedure Test
+
+1. Stop service
+2. Restore data from backup
+3. Start service
+4. Verify functionality
+5. Check logs for errors
+
+---
+
+## Performance Testing
+
+### Load Testing
+
+```bash
+# Using hey or ab
+hey -n 1000 -c 10 https://service.vish.gg
+
+# Check response times
+curl -w "@curl-format.txt" -o /dev/null -s https://service.vish.gg
+
+# curl-format.txt:
+# time_namelookup:  %{time_namelookup}\n
+# time_connect:     %{time_connect}\n
+# time_appconnect:  %{time_appconnect}\n
+# time_redirect:    %{time_redirect}\n
+# time_pretransfer: %{time_pretransfer}\n
+# time_starttransfer: %{time_starttransfer}\n
+# time_total:       %{time_total}\n
+```
+
+### Resource Testing
+
+```bash
+# Monitor during load
+docker stats --no-stream
+
+# Check for OOM kills
+dmesg | grep -i "out of memory"
+
+# Monitor disk I/O
+iostat -x 1
+```
+
+---
+
+## Security Testing
+
+### Vulnerability Scanning
+
+```bash
+# Trivy scan
+trivy image --severity HIGH,CRITICAL <image>
+
+# Check for secrets
+trivy fs --security-checks secrets /path/to/compose
+
+# Docker scan
+docker scan <image>
+```
+
+### SSL/TLS Testing
+
+```bash
+# SSL Labs
+# Visit: https://www.ssllabs.com/ssltest/
+
+# CLI check
+openssl s_client -connect service.vish.gg:443
+
+# Check certificates
+certinfo service.vish.gg
+```
+
+---
+
+## Network Testing
+
+### Connectivity
+
+```bash
+# Port scan
+nmap -p 1-1000 192.168.0.x
+
+# DNS check
+dig service.vish.local
+nslookup service.vish.local
+
+# traceroute
+traceroute service.vish.gg
+```
+
+### Firewall Testing
+
+```bash
+# Check open ports
+ss -tulpn
+
+# Test from outside
+# Use online port scanner
+
+# Test blocked access
+curl -I http://internal-service:port
+# Should fail without VPN
+```
+
+---
+
+## Regression Testing
+
+### After Updates
+
+1. Check service starts
+2. Verify all features
+3. Test SSO if enabled
+4. Check monitoring
+5. Verify backups
+
+### Critical Path Tests
+
+| Path | Steps |
+|------|-------|
+| External access | VPN → NPM → Service |
+| SSO login | Service → Auth → Dashboard |
+| Media playback | Request → Download → Play |
+| Backup restore | Stop → Restore → Verify → Start |
+
+---
+
+## Acceptance Criteria
+
+### New Service
+
+- [ ] Starts without errors
+- [ ] UI accessible
+- [ ] Basic function works
+- [ ] SSO configured (if supported)
+- [ ] Monitoring enabled
+- [ ] Backup configured
+- [ ] Documentation created
+
+### Infrastructure Change
+
+- [ ] All services running
+- [ ] No new alerts
+- [ ] Monitoring healthy
+- [ ] Backups completed
+- [ ] Users notified (if needed)
+
+---
+
+## Links
+
+- [Monitoring Architecture](../infrastructure/MONITORING_ARCHITECTURE.md)
+- [Backup Procedures](../BACKUP_PROCEDURES.md)
+- [Disaster Recovery](../troubleshooting/disaster-recovery.md)