304 lines
5.0 KiB
Markdown
304 lines
5.0 KiB
Markdown
# Testing Procedures
|
|
|
|
*Testing guidelines for the homelab infrastructure*
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
This document outlines testing procedures for deploying new services, making infrastructure changes, and validating functionality.
|
|
|
|
---
|
|
|
|
## Pre-Deployment Testing
|
|
|
|
### New Service Checklist
|
|
|
|
- [ ] Review Docker image (official, stars, updates)
|
|
- [ ] Check for security vulnerabilities
|
|
- [ ] Verify resource requirements
|
|
- [ ] Test locally first
|
|
- [ ] Verify compose syntax
|
|
- [ ] Check port availability
|
|
- [ ] Test volume paths
|
|
|
|
### Compose Validation
|
|
|
|
```bash
|
|
# Validate syntax
|
|
docker-compose config --quiet
|
|
|
|
# Check for errors
|
|
docker-compose up --dry-run
|
|
|
|
# Pull images
|
|
docker-compose pull
|
|
```
|
|
|
|
---
|
|
|
|
## Local Testing
|
|
|
|
### Docker Desktop / Mini Setup
|
|
|
|
1. Create test compose file
|
|
2. Run on local machine
|
|
3. Verify all features work
|
|
4. Document any issues
|
|
|
|
### Test Environment
|
|
|
|
If available, use staging:
|
|
- Staging host: `seattle` VM
|
|
- Test domain: `*.test.vish.local`
|
|
- Shared internally only
|
|
|
|
---
|
|
|
|
## Integration Testing
|
|
|
|
### Authentik SSO
|
|
|
|
```bash
|
|
# Test login flow
|
|
1. Open service
|
|
2. Click "Login with Authentik"
|
|
3. Verify redirect to Authentik
|
|
4. Enter credentials
|
|
5. Verify return to service
|
|
6. Check user profile
|
|
```
|
|
|
|
### Nginx Proxy Manager
|
|
|
|
```bash
|
|
# Test proxy host
|
|
curl -H "Host: service.vish.local" http://localhost
|
|
|
|
# Test SSL
|
|
curl -k https://service.vish.gg
|
|
|
|
# Check headers
|
|
curl -I https://service.vish.gg
|
|
```
|
|
|
|
### Database Connections
|
|
|
|
```bash
|
|
# PostgreSQL
|
|
docker exec <container> psql -U user -c "SELECT 1"
|
|
|
|
# Test from application
|
|
docker exec <app> nc -zv db 5432
|
|
```
|
|
|
|
---
|
|
|
|
## Monitoring Validation
|
|
|
|
### Prometheus Targets
|
|
|
|
1. Open Prometheus UI
|
|
2. Go to Status → Targets
|
|
3. Verify all targets are UP
|
|
4. Check for scrape errors
|
|
|
|
### Alert Testing
|
|
|
|
```bash
|
|
# Trigger test alert
|
|
curl -X POST http://alertmanager:9093/api/v1/alerts \
|
|
-H "Content-Type: application/json" \
|
|
-d '[{
|
|
"labels": {
|
|
"alertname": "TestAlert",
|
|
"severity": "critical"
|
|
},
|
|
"annotations": {
|
|
"summary": "Test alert"
|
|
}
|
|
}]'
|
|
```
|
|
|
|
### Grafana Dashboards
|
|
|
|
- [ ] All panels load
|
|
- [ ] Data populates
|
|
- [ ] No errors in console
|
|
- [ ] Alerts configured
|
|
|
|
---
|
|
|
|
## Backup Testing
|
|
|
|
### Full Backup Test
|
|
|
|
```bash
|
|
# Run backup
|
|
ansible-playbook ansible/automation/playbooks/backup_configs.yml
|
|
ansible-playbook ansible/automation/playbooks/backup_databases.yml
|
|
|
|
# Verify backup files exist
|
|
ls -la /backup/
|
|
|
|
# Test restore to test environment
|
|
# (do NOT overwrite production!)
|
|
```
|
|
|
|
### Restore Procedure Test
|
|
|
|
1. Stop service
|
|
2. Restore data from backup
|
|
3. Start service
|
|
4. Verify functionality
|
|
5. Check logs for errors
|
|
|
|
---
|
|
|
|
## Performance Testing
|
|
|
|
### Load Testing
|
|
|
|
```bash
|
|
# Using hey or ab
|
|
hey -n 1000 -c 10 https://service.vish.gg
|
|
|
|
# Check response times
|
|
curl -w "@curl-format.txt" -o /dev/null -s https://service.vish.gg
|
|
|
|
# curl-format.txt:
|
|
# time_namelookup: %{time_namelookup}\n
|
|
# time_connect: %{time_connect}\n
|
|
# time_appconnect: %{time_appconnect}\n
|
|
# time_redirect: %{time_redirect}\n
|
|
# time_pretransfer: %{time_pretransfer}\n
|
|
# time_starttransfer: %{time_starttransfer}\n
|
|
# time_total: %{time_total}\n
|
|
```
|
|
|
|
### Resource Testing
|
|
|
|
```bash
|
|
# Monitor during load
|
|
docker stats --no-stream
|
|
|
|
# Check for OOM kills
|
|
dmesg | grep -i "out of memory"
|
|
|
|
# Monitor disk I/O
|
|
iostat -x 1
|
|
```
|
|
|
|
---
|
|
|
|
## Security Testing
|
|
|
|
### Vulnerability Scanning
|
|
|
|
```bash
|
|
# Trivy scan
|
|
trivy image --severity HIGH,CRITICAL <image>
|
|
|
|
# Check for secrets
|
|
trivy fs --security-checks secrets /path/to/compose
|
|
|
|
# Docker scan
|
|
docker scan <image>
|
|
```
|
|
|
|
### SSL/TLS Testing
|
|
|
|
```bash
|
|
# SSL Labs
|
|
# Visit: https://www.ssllabs.com/ssltest/
|
|
|
|
# CLI check
|
|
openssl s_client -connect service.vish.gg:443
|
|
|
|
# Check certificates
|
|
certinfo service.vish.gg
|
|
```
|
|
|
|
---
|
|
|
|
## Network Testing
|
|
|
|
### Connectivity
|
|
|
|
```bash
|
|
# Port scan
|
|
nmap -p 1-1000 192.168.0.x
|
|
|
|
# DNS check
|
|
dig service.vish.local
|
|
nslookup service.vish.local
|
|
|
|
# traceroute
|
|
traceroute service.vish.gg
|
|
```
|
|
|
|
### Firewall Testing
|
|
|
|
```bash
|
|
# Check open ports
|
|
ss -tulpn
|
|
|
|
# Test from outside
|
|
# Use online port scanner
|
|
|
|
# Test blocked access
|
|
curl -I http://internal-service:port
|
|
# Should fail without VPN
|
|
```
|
|
|
|
---
|
|
|
|
## Regression Testing
|
|
|
|
### After Updates
|
|
|
|
1. Check service starts
|
|
2. Verify all features
|
|
3. Test SSO if enabled
|
|
4. Check monitoring
|
|
5. Verify backups
|
|
|
|
### Critical Path Tests
|
|
|
|
| Path | Steps |
|
|
|------|-------|
|
|
| External access | VPN → NPM → Service |
|
|
| SSO login | Service → Auth → Dashboard |
|
|
| Media playback | Request → Download → Play |
|
|
| Backup restore | Stop → Restore → Verify → Start |
|
|
|
|
---
|
|
|
|
## Acceptance Criteria
|
|
|
|
### New Service
|
|
|
|
- [ ] Starts without errors
|
|
- [ ] UI accessible
|
|
- [ ] Basic function works
|
|
- [ ] SSO configured (if supported)
|
|
- [ ] Monitoring enabled
|
|
- [ ] Backup configured
|
|
- [ ] Documentation created
|
|
|
|
### Infrastructure Change
|
|
|
|
- [ ] All services running
|
|
- [ ] No new alerts
|
|
- [ ] Monitoring healthy
|
|
- [ ] Backups completed
|
|
- [ ] Users notified (if needed)
|
|
|
|
---
|
|
|
|
## Links
|
|
|
|
- [Monitoring Architecture](../infrastructure/MONITORING_ARCHITECTURE.md)
|
|
- [Backup Procedures](../BACKUP_PROCEDURES.md)
|
|
- [Disaster Recovery](../troubleshooting/disaster-recovery.md)
|