# Troubleshooting Guide

This guide covers common issues you might encounter when deploying and running the Arrs media stack, along with step-by-step solutions.

## Table of Contents

1. [Quick Diagnostics](#quick-diagnostics)
2. [Deployment Issues](#deployment-issues)
3. [Service Issues](#service-issues)
4. [Network and Connectivity](#network-and-connectivity)
5. [Storage and Permissions](#storage-and-permissions)
6. [VPN Issues](#vpn-issues)
7. [Performance Issues](#performance-issues)
8. [Backup and Recovery](#backup-and-recovery)
9. [Getting Help](#getting-help)

## Quick Diagnostics

### Essential Commands

```bash
# Check all container status
docker ps -a

# Check system resources
htop
df -h

# Check logs for all services
docker-compose logs --tail=50

# Check specific service logs
docker logs [service_name] --tail=50

# Test network connectivity
ping google.com
curl -I https://google.com
```

### Health Check Script

Create a quick health check script:

```bash
#!/bin/bash
# Save as ~/check_health.sh

echo "=== System Resources ==="
df -h | grep -E "(Filesystem|/dev/)"
free -h
echo ""

echo "=== Docker Status ==="
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
echo ""

echo "=== Service Health ==="
services=("sonarr" "radarr" "lidarr" "bazarr" "prowlarr" "qbittorrent" "plex" "tubearchivist" "tubearchivist-es" "tubearchivist-redis" "jellyseerr" "tautulli")
for service in "${services[@]}"; do
    if docker ps | grep -q $service; then
        echo "✅ $service: Running"
    else
        echo "❌ $service: Not running"
    fi
done
```

Make it executable and run:
```bash
chmod +x ~/check_health.sh
~/check_health.sh
```

## Deployment Issues

### Ansible Playbook Fails

#### Issue: "Permission denied" errors

**Symptoms:**
```
TASK [Create docker user] ****
fatal: [localhost]: FAILED! => {"msg": "Permission denied"}
```

**Solutions:**
1. Ensure you're running as a user with sudo privileges:
   ```bash
   sudo -v  # Test sudo access
   ```

2. If using a non-root user, ensure they're in the sudo group:
   ```bash
   sudo usermod -aG sudo $USER
   # Log out and back in
   ```

3. Run with explicit sudo if needed:
   ```bash
   ansible-playbook -i inventory/hosts site.yml --become --ask-become-pass
   ```

#### Issue: "Docker not found" after installation

**Symptoms:**
```
TASK [Start Docker service] ****
fatal: [localhost]: FAILED! => {"msg": "Could not find the requested service docker"}
```

**Solutions:**
1. Manually install Docker:
   ```bash
   curl -fsSL https://get.docker.com -o get-docker.sh
   sudo sh get-docker.sh
   sudo usermod -aG docker $USER
   # Log out and back in
   ```

2. Restart the deployment:
   ```bash
   ansible-playbook -i inventory/hosts site.yml
   ```

#### Issue: "Port already in use"

**Symptoms:**
```
ERROR: for sonarr  Cannot start service sonarr: driver failed programming external connectivity on endpoint sonarr: Bind for 0.0.0.0:8989 failed: port is already allocated
```

**Solutions:**
1. Check what's using the port:
   ```bash
   sudo netstat -tulpn | grep :8989
   sudo lsof -i :8989
   ```

2. Stop the conflicting service or change ports in `group_vars/all.yml`:
   ```yaml
   ports:
     sonarr: 8990  # Changed from 8989
   ```

3. Redeploy:
   ```bash
   ansible-playbook -i inventory/hosts site.yml
   ```

### Configuration File Issues

#### Issue: YAML syntax errors

**Symptoms:**
```
ERROR! Syntax Error while loading YAML.
```

**Solutions:**
1. Validate YAML syntax:
   ```bash
   python3 -c "import yaml; yaml.safe_load(open('group_vars/all.yml'))"
   ```

2. Common YAML mistakes:
   - Missing quotes around special characters
   - Incorrect indentation (use spaces, not tabs)
   - Missing colons after keys

3. Use a YAML validator online or in your editor

#### Issue: Undefined variables

**Symptoms:**
```
AnsibleUndefinedVariable: 'vpn_provider' is undefined
```

**Solutions:**
1. Check if variable is defined in `group_vars/all.yml`
2. Ensure proper indentation and spelling
3. Add missing variables:
   ```yaml
   vpn_provider: "nordvpn"  # Add if missing
   ```

## Service Issues

### Services Won't Start

#### Issue: Container exits immediately

**Check logs:**
```bash
docker logs [container_name]
```

**Common causes and solutions:**

1. **Permission issues:**
   ```bash
   # Fix ownership
   sudo chown -R 1000:1000 /home/arrs/docker
   sudo chown -R 1000:1000 /home/arrs/media
   ```

2. **Missing directories:**
   ```bash
   # Create missing directories
   mkdir -p /home/arrs/media/{movies,tv,music,downloads}
   mkdir -p /home/arrs/docker/{sonarr,radarr,lidarr,bazarr,prowlarr,qbittorrent}
   ```

3. **Configuration file corruption:**
   ```bash
   # Remove corrupted config and restart
   sudo rm -rf /home/arrs/docker/sonarr/config.xml
   docker restart sonarr
   ```

#### Issue: Service starts but web interface not accessible

**Symptoms:**
- Container shows as running
- Can't access web interface

**Solutions:**
1. Check if service is listening on correct port:
   ```bash
   docker exec sonarr netstat -tulpn | grep :8989
   ```

2. Check firewall:
   ```bash
   sudo ufw status
   sudo ufw allow 8989  # Open required port
   ```

3. Check if service is bound to localhost only:
   ```bash
   docker logs sonarr | grep -i "listening\|bind"
   ```

### Database Issues

#### Issue: Sonarr/Radarr database corruption

**Symptoms:**
- Service won't start
- Logs show database errors
- Web interface shows errors

**Solutions:**
1. **Backup current database:**
   ```bash
   cp /home/arrs/docker/sonarr/sonarr.db /home/arrs/docker/sonarr/sonarr.db.backup
   ```

2. **Try database repair:**
   ```bash
   # Install sqlite3
   sudo apt install sqlite3
   
   # Check database integrity
   sqlite3 /home/arrs/docker/sonarr/sonarr.db "PRAGMA integrity_check;"
   
   # Repair if needed
   sqlite3 /home/arrs/docker/sonarr/sonarr.db ".recover" | sqlite3 /home/arrs/docker/sonarr/sonarr_recovered.db
   ```

3. **Restore from backup:**
   ```bash
   # Stop service
   docker stop sonarr
   
   # Restore backup
   cp /home/arrs/docker/sonarr/sonarr.db.backup /home/arrs/docker/sonarr/sonarr.db
   
   # Start service
   docker start sonarr
   ```

## Network and Connectivity

### Can't Access Services from Outside

#### Issue: Services only accessible from localhost

**Solutions:**
1. **Check Docker network configuration:**
   ```bash
   docker network ls
   docker network inspect arrs_network
   ```

2. **Verify port bindings:**
   ```bash
   docker port sonarr
   ```

3. **Check VPS firewall:**
   ```bash
   # Ubuntu/Debian
   sudo ufw status
   sudo ufw allow 8989
   
   # CentOS/RHEL
   sudo firewall-cmd --list-ports
   sudo firewall-cmd --add-port=8989/tcp --permanent
   sudo firewall-cmd --reload
   ```

4. **Check cloud provider firewall:**
   - AWS: Security Groups
   - DigitalOcean: Cloud Firewalls
   - Google Cloud: VPC Firewall Rules

### DNS Resolution Issues

#### Issue: Services can't resolve external domains

**Symptoms:**
- Can't download metadata
- Indexer tests fail
- Updates don't work

**Solutions:**
1. **Check DNS in containers:**
   ```bash
   docker exec sonarr nslookup google.com
   docker exec sonarr cat /etc/resolv.conf
   ```

2. **Fix DNS configuration:**
   ```bash
   # Edit Docker daemon configuration
   sudo nano /etc/docker/daemon.json
   ```
   
   Add:
   ```json
   {
     "dns": ["8.8.8.8", "1.1.1.1"]
   }
   ```
   
   Restart Docker:
   ```bash
   sudo systemctl restart docker
   docker-compose up -d
   ```

## Storage and Permissions

### Permission Denied Errors

#### Issue: Services can't write to media directories

**Symptoms:**
- Downloads fail
- Can't move files
- Import errors

**Solutions:**
1. **Check current permissions:**
   ```bash
   ls -la /home/arrs/media/
   ls -la /home/arrs/docker/
   ```

2. **Fix ownership:**
   ```bash
   # Get user/group IDs
   id arrs
   
   # Fix ownership (replace 1000:1000 with actual IDs)
   sudo chown -R 1000:1000 /home/arrs/media
   sudo chown -R 1000:1000 /home/arrs/docker
   ```

3. **Fix permissions:**
   ```bash
   sudo chmod -R 755 /home/arrs/media
   sudo chmod -R 755 /home/arrs/docker
   ```

4. **Verify container user mapping:**
   ```bash
   docker exec sonarr id
   ```

### Disk Space Issues

#### Issue: No space left on device

**Solutions:**
1. **Check disk usage:**
   ```bash
   df -h
   du -sh /home/arrs/* | sort -hr
   ```

2. **Clean up Docker:**
   ```bash
   # Remove unused containers, networks, images
   docker system prune -a
   
   # Remove unused volumes (CAREFUL!)
   docker volume prune
   ```

3. **Clean up logs:**
   ```bash
   # Truncate large log files
   sudo truncate -s 0 /var/log/syslog
   sudo truncate -s 0 /var/log/kern.log
   
   # Clean Docker logs
   docker logs sonarr 2>/dev/null | wc -l  # Check log size
   sudo sh -c 'echo "" > $(docker inspect --format="{{.LogPath}}" sonarr)'
   ```

4. **Move media to external storage:**
   ```bash
   # Mount additional storage
   sudo mkdir /mnt/media
   sudo mount /dev/sdb1 /mnt/media
   
   # Update configuration
   nano group_vars/all.yml
   # Change media_root: "/mnt/media"
   ```

## TubeArchivist Issues

### TubeArchivist Won't Start

#### Issue: Elasticsearch container fails to start

**Symptoms:**
- TubeArchivist shows "Elasticsearch connection failed"
- Elasticsearch container exits with memory errors
- Web interface shows database connection errors

**Solutions:**
1. **Increase memory allocation:**
   ```bash
   # Check available memory
   free -h
   
   # If less than 4GB available, reduce ES memory in docker-compose.yml
   # Change ES_JAVA_OPTS from -Xms1g -Xmx1g to -Xms512m -Xmx512m
   ```

2. **Fix Elasticsearch permissions:**
   ```bash
   sudo chown -R 1000:1000 /home/arrs/docker/tubearchivist/es
   sudo chmod -R 755 /home/arrs/docker/tubearchivist/es
   ```

3. **Check disk space:**
   ```bash
   df -h /home/arrs/docker/tubearchivist/
   ```

#### Issue: Downloads fail or get stuck

**Symptoms:**
- Videos remain in "Pending" status
- Download queue shows errors
- yt-dlp errors in logs

**Solutions:**
1. **Update yt-dlp:**
   ```bash
   docker exec tubearchivist pip install --upgrade yt-dlp
   docker restart tubearchivist
   ```

2. **Check YouTube channel/video availability:**
   - Verify the channel/video is still available
   - Check if the channel has geographic restrictions
   - Try downloading a different video to test

3. **Clear download queue:**
   ```bash
   # Access TubeArchivist web interface
   # Go to Downloads → Queue → Clear Failed
   ```

4. **Check storage space:**
   ```bash
   df -h /home/arrs/media/youtube
   ```

#### Issue: Videos won't play or thumbnails missing

**Symptoms:**
- Videos show but won't play
- Missing thumbnails
- Playback errors

**Solutions:**
1. **Check file permissions:**
   ```bash
   ls -la /home/arrs/media/youtube/
   sudo chown -R 1000:1000 /home/arrs/media/youtube
   ```

2. **Regenerate thumbnails:**
   - Go to Settings → Application → Reindex
   - Select "Thumbnails" and run reindex

3. **Check video file integrity:**
   ```bash
   # Test video files
   find /home/arrs/media/youtube -name "*.mp4" -exec file {} \;
   ```

### TubeArchivist Performance Issues

#### Issue: Slow downloads or high CPU usage

**Solutions:**
1. **Limit concurrent downloads:**
   - Go to Settings → Download → Max concurrent downloads
   - Reduce from default (4) to 1-2 for lower-end systems

2. **Adjust video quality:**
   - Go to Settings → Download → Video quality
   - Choose lower quality (720p instead of 1080p) to reduce processing

3. **Schedule downloads during off-peak hours:**
   - Go to Settings → Scheduling
   - Set download windows for low-usage periods

#### Issue: Database performance problems

**Solutions:**
1. **Optimize Elasticsearch:**
   ```bash
   # Increase refresh interval
   docker exec tubearchivist-es curl -X PUT "localhost:9200/_settings" -H 'Content-Type: application/json' -d'
   {
     "index": {
       "refresh_interval": "30s"
     }
   }'
   ```

2. **Clean up old data:**
   - Go to Settings → Application → Cleanup
   - Remove old downloads and unused thumbnails

## VPN Issues

### VPN Won't Connect

#### Issue: Gluetun fails to connect

**Check logs:**
```bash
docker logs gluetun --tail=50
```

**Common solutions:**

1. **Wrong credentials:**
   ```yaml
   # Verify in group_vars/all.yml
   openvpn_user: "correct_username"
   openvpn_password: "correct_password"
   ```

2. **Unsupported server location:**
   ```yaml
   # Try different countries
   vpn_countries: "Netherlands,Germany"
   ```

3. **Provider API issues:**
   ```bash
   # Try manual server selection
   vpn_server_hostnames: "nl123.nordvpn.com"
   ```

#### Issue: qBittorrent can't connect through VPN

**Solutions:**
1. **Check network mode:**
   ```bash
   docker inspect qbittorrent | grep NetworkMode
   # Should show: "NetworkMode": "service:gluetun"
   ```

2. **Test connectivity:**
   ```bash
   # Test from qBittorrent container
   docker exec qbittorrent curl -s https://ipinfo.io/ip
   ```

3. **Restart VPN stack:**
   ```bash
   docker restart gluetun
   docker restart qbittorrent
   ```

## Performance Issues

### Slow Performance

#### Issue: Services are slow or unresponsive

**Solutions:**
1. **Check system resources:**
   ```bash
   htop
   iotop  # Install with: sudo apt install iotop
   ```

2. **Optimize Docker:**
   ```bash
   # Limit container resources in docker-compose.yml
   services:
     sonarr:
       deploy:
         resources:
           limits:
             memory: 512M
           reservations:
             memory: 256M
   ```

3. **Optimize database:**
   ```bash
   # Vacuum SQLite databases
   sqlite3 /home/arrs/docker/sonarr/sonarr.db "VACUUM;"
   sqlite3 /home/arrs/docker/radarr/radarr.db "VACUUM;"
   ```

4. **Check disk I/O:**
   ```bash
   # Test disk speed
   dd if=/dev/zero of=/tmp/test bs=1M count=1000
   rm /tmp/test
   ```

### High CPU Usage

#### Issue: Containers using too much CPU

**Solutions:**
1. **Identify problematic containers:**
   ```bash
   docker stats
   ```

2. **Check for runaway processes:**
   ```bash
   docker exec sonarr ps aux
   ```

3. **Limit CPU usage:**
   ```yaml
   # In docker-compose.yml
   services:
     sonarr:
       deploy:
         resources:
           limits:
             cpus: '0.5'  # Limit to 50% of one CPU core
   ```

## Backup and Recovery

### Backup Issues

#### Issue: Backup script fails

**Check backup logs:**
```bash
# Check if backup script exists
ls -la /home/arrs/scripts/backup.sh

# Check backup logs
tail -f /var/log/arrs-backup.log
```

**Solutions:**
1. **Fix permissions:**
   ```bash
   chmod +x /home/arrs/scripts/backup.sh
   ```

2. **Test backup manually:**
   ```bash
   sudo /home/arrs/scripts/backup.sh
   ```

3. **Check backup destination:**
   ```bash
   df -h /home/arrs/backups
   ```

### Recovery Procedures

#### Issue: Need to restore from backup

**Steps:**
1. **Stop all services:**
   ```bash
   docker-compose down
   ```

2. **Restore configuration:**
   ```bash
   # Extract backup
   tar -xzf /home/arrs/backups/arrs-backup-YYYYMMDD.tar.gz -C /home/arrs/
   ```

3. **Fix permissions:**
   ```bash
   sudo chown -R 1000:1000 /home/arrs/docker
   ```

4. **Start services:**
   ```bash
   docker-compose up -d
   ```

## Getting Help

### Before Asking for Help

1. **Check logs:**
   ```bash
   docker logs [service_name] --tail=100
   ```

2. **Gather system information:**
   ```bash
   # Create debug info
   echo "=== System Info ===" > debug.txt
   uname -a >> debug.txt
   docker --version >> debug.txt
   docker-compose --version >> debug.txt
   
   echo "=== Container Status ===" >> debug.txt
   docker ps -a >> debug.txt
   
   echo "=== Disk Usage ===" >> debug.txt
   df -h >> debug.txt
   
   echo "=== Memory Usage ===" >> debug.txt
   free -h >> debug.txt
   ```

3. **Sanitize sensitive information:**
   - Remove passwords, API keys, personal paths
   - Replace IP addresses with placeholders

### Where to Get Help

1. **GitHub Issues**: Create an issue with:
   - Clear description of the problem
   - Steps to reproduce
   - Error messages
   - System information
   - Configuration (sanitized)

2. **Community Forums**:
   - r/selfhosted
   - r/sonarr, r/radarr
   - Discord servers for specific applications

3. **Documentation**:
   - Official documentation for each service
   - Docker documentation
   - Ansible documentation

### Common Support Questions

**Q: "It doesn't work"**
A: Please provide specific error messages and logs.

**Q: "Can't access from outside"**
A: Check firewall settings and port configurations.

**Q: "Downloads don't start"**
A: Check indexer configuration and download client settings.

**Q: "VPN not working"**
A: Verify credentials and check Gluetun logs.

**Q: "Running out of space"**
A: Clean up Docker images and logs, consider external storage.

### Emergency Recovery

If everything breaks:

1. **Stop all services:**
   ```bash
   docker-compose down
   ```

2. **Backup current state:**
   ```bash
   sudo tar -czf emergency-backup-$(date +%Y%m%d).tar.gz /home/arrs/docker
   ```

3. **Reset to clean state:**
   ```bash
   # Remove all containers and volumes
   docker system prune -a --volumes
   
   # Redeploy
   ansible-playbook -i inventory/hosts site.yml
   ```

4. **Restore data from backups if needed**

Remember: Most issues are fixable with patience and systematic troubleshooting. Don't panic, and always backup before making major changes!