# Stoatchat Service Management

Complete guide for managing Stoatchat services on the Seattle VM.

## Service Architecture

```
Stoatchat Services (Native Binaries)
├── revolt-delta (API Server) → Port 14702
├── revolt-bonfire (Events WebSocket) → Port 14703
├── revolt-autumn (File Server) → Port 14704
├── revolt-january (Media Proxy) → Port 14705
└── revolt-gifbox (GIF Service) → Port 14706

Infrastructure Services (Docker)
├── Redis (KeyDB) → Port 6380
├── MongoDB → Port 27017
├── MinIO → Port 14009
└── LiveKit → Port 7880

System Services
└── Nginx → Ports 80, 443
```

## Starting Services

### 1. Start Infrastructure Services
```bash
cd /root/stoatchat

# Start all Docker services
docker-compose up -d

# Check status
docker-compose ps

# Wait for services to be ready (important!)
sleep 30
```

### 2. Start Stoatchat Services
```bash
cd /root/stoatchat

# Start all services in background
nohup ./target/debug/revolt-delta > api.log 2>&1 &
nohup ./target/debug/revolt-bonfire > events.log 2>&1 &
nohup ./target/debug/revolt-autumn > files.log 2>&1 &
nohup ./target/debug/revolt-january > proxy.log 2>&1 &
nohup ./target/debug/revolt-gifbox > gifbox.log 2>&1 &

echo "All Stoatchat services started"
```

### 3. Automated Startup Script
```bash
# Create startup script
cat > /root/stoatchat/start-all-services.sh << 'EOF'
#!/bin/bash
cd /root/stoatchat

echo "Starting infrastructure services..."
docker-compose up -d

echo "Waiting for infrastructure to be ready..."
sleep 30

echo "Starting Stoatchat services..."
nohup ./target/debug/revolt-delta > api.log 2>&1 &
nohup ./target/debug/revolt-bonfire > events.log 2>&1 &
nohup ./target/debug/revolt-autumn > files.log 2>&1 &
nohup ./target/debug/revolt-january > proxy.log 2>&1 &
nohup ./target/debug/revolt-gifbox > gifbox.log 2>&1 &

echo "All services started. Checking status..."
sleep 5
ps aux | grep revolt | grep -v grep
EOF

chmod +x /root/stoatchat/start-all-services.sh
```

## Stopping Services

### 1. Stop Stoatchat Services
```bash
# Stop all revolt processes
pkill -f revolt-

# Or stop individually
pkill -f revolt-delta      # API
pkill -f revolt-bonfire    # Events
pkill -f revolt-autumn     # Files
pkill -f revolt-january    # Proxy
pkill -f revolt-gifbox     # GIF
```

### 2. Stop Infrastructure Services
```bash
cd /root/stoatchat

# Stop all Docker services
docker-compose down

# Or stop individually
docker-compose stop redis
docker-compose stop database
docker-compose stop minio
docker-compose stop livekit
```

### 3. Complete Shutdown Script
```bash
# Create shutdown script
cat > /root/stoatchat/stop-all-services.sh << 'EOF'
#!/bin/bash
cd /root/stoatchat

echo "Stopping Stoatchat services..."
pkill -f revolt-

echo "Stopping infrastructure services..."
docker-compose down

echo "All services stopped."
EOF

chmod +x /root/stoatchat/stop-all-services.sh
```

## Restarting Services

### 1. Restart Individual Stoatchat Service
```bash
cd /root/stoatchat

# Example: Restart API server
pkill -f revolt-delta
nohup ./target/debug/revolt-delta > api.log 2>&1 &

# Example: Restart Events service
pkill -f revolt-bonfire
nohup ./target/debug/revolt-bonfire > events.log 2>&1 &
```

### 2. Restart Infrastructure Service
```bash
cd /root/stoatchat

# Example: Restart Redis
docker-compose restart redis

# Example: Restart MongoDB
docker-compose restart database
```

### 3. Complete Restart
```bash
cd /root/stoatchat

# Stop everything
./stop-all-services.sh

# Wait a moment
sleep 5

# Start everything
./start-all-services.sh
```

## Service Status Monitoring

### 1. Check Running Processes
```bash
# Check all Stoatchat processes
ps aux | grep revolt | grep -v grep

# Check specific service
ps aux | grep revolt-delta

# Check with process tree
pstree -p | grep revolt
```

### 2. Check Listening Ports
```bash
# Check all Stoatchat ports
ss -tlnp | grep -E "(14702|14703|14704|14705|14706|7880)"

# Check specific port
ss -tlnp | grep 14702

# Check with netstat
netstat -tlnp | grep -E "(14702|14703|14704|14705|14706|7880)"
```

### 3. Check Docker Services
```bash
cd /root/stoatchat

# Check all services
docker-compose ps

# Check specific service
docker-compose ps redis

# Check service logs
docker-compose logs redis
docker-compose logs database
docker-compose logs minio
docker-compose logs livekit
```

### 4. Service Health Check
```bash
# Test all endpoints
curl -s https://api.st.vish.gg/ | jq .revolt
curl -s https://files.st.vish.gg/ | jq .autumn
curl -s https://proxy.st.vish.gg/ | jq .january
curl -s https://voice.st.vish.gg/

# Or use the health check script
/root/stoatchat/health-check.sh
```

## Log Management

### 1. View Service Logs
```bash
cd /root/stoatchat

# View current logs
tail -f api.log        # API server
tail -f events.log     # Events WebSocket
tail -f files.log      # File server
tail -f proxy.log      # Media proxy
tail -f gifbox.log     # GIF service

# View all logs simultaneously
tail -f *.log

# View with timestamps
tail -f api.log | while read line; do echo "$(date): $line"; done
```

### 2. Log Rotation
```bash
# Create log rotation script
cat > /root/stoatchat/rotate-logs.sh << 'EOF'
#!/bin/bash
cd /root/stoatchat

# Rotate logs if they're larger than 100MB
for log in api.log events.log files.log proxy.log gifbox.log; do
    if [ -f "$log" ] && [ $(stat -f%z "$log" 2>/dev/null || stat -c%s "$log") -gt 104857600 ]; then
        mv "$log" "$log.$(date +%Y%m%d-%H%M%S)"
        touch "$log"
        echo "Rotated $log"
    fi
done
EOF

chmod +x /root/stoatchat/rotate-logs.sh

# Add to crontab for daily rotation
# crontab -e
# 0 2 * * * /root/stoatchat/rotate-logs.sh
```

### 3. Clear Logs
```bash
cd /root/stoatchat

# Clear all logs
> api.log
> events.log
> files.log
> proxy.log
> gifbox.log

# Or remove and recreate
rm -f *.log
touch api.log events.log files.log proxy.log gifbox.log
```

## Configuration Management

### 1. Backup Configuration
```bash
cd /root/stoatchat

# Create backup
cp Revolt.overrides.toml Revolt.overrides.toml.backup.$(date +%Y%m%d)
cp livekit.yml livekit.yml.backup.$(date +%Y%m%d)
cp compose.yml compose.yml.backup.$(date +%Y%m%d)
```

### 2. Apply Configuration Changes
```bash
cd /root/stoatchat

# After editing Revolt.overrides.toml
# Restart affected services
pkill -f revolt-
./start-all-services.sh

# After editing livekit.yml
docker-compose restart livekit

# After editing compose.yml
docker-compose down
docker-compose up -d
```

### 3. Validate Configuration
```bash
cd /root/stoatchat

# Check TOML syntax
python3 -c "import toml; toml.load('Revolt.overrides.toml')" && echo "TOML valid"

# Check YAML syntax
python3 -c "import yaml; yaml.safe_load(open('livekit.yml'))" && echo "YAML valid"
python3 -c "import yaml; yaml.safe_load(open('compose.yml'))" && echo "Compose valid"

# Check nginx configuration
nginx -t
```

## Systemd Service Setup (Optional)

### 1. Create Systemd Services
```bash
# API Service
cat > /etc/systemd/system/stoatchat-api.service << 'EOF'
[Unit]
Description=Stoatchat API Server
After=network.target docker.service
Requires=docker.service

[Service]
Type=simple
User=root
WorkingDirectory=/root/stoatchat
ExecStart=/root/stoatchat/target/debug/revolt-delta
Restart=always
RestartSec=10
StandardOutput=append:/root/stoatchat/api.log
StandardError=append:/root/stoatchat/api.log

[Install]
WantedBy=multi-user.target
EOF

# Events Service
cat > /etc/systemd/system/stoatchat-events.service << 'EOF'
[Unit]
Description=Stoatchat Events WebSocket
After=network.target docker.service stoatchat-api.service
Requires=docker.service

[Service]
Type=simple
User=root
WorkingDirectory=/root/stoatchat
ExecStart=/root/stoatchat/target/debug/revolt-bonfire
Restart=always
RestartSec=10
StandardOutput=append:/root/stoatchat/events.log
StandardError=append:/root/stoatchat/events.log

[Install]
WantedBy=multi-user.target
EOF

# Files Service
cat > /etc/systemd/system/stoatchat-files.service << 'EOF'
[Unit]
Description=Stoatchat File Server
After=network.target docker.service
Requires=docker.service

[Service]
Type=simple
User=root
WorkingDirectory=/root/stoatchat
ExecStart=/root/stoatchat/target/debug/revolt-autumn
Restart=always
RestartSec=10
StandardOutput=append:/root/stoatchat/files.log
StandardError=append:/root/stoatchat/files.log

[Install]
WantedBy=multi-user.target
EOF

# Proxy Service
cat > /etc/systemd/system/stoatchat-proxy.service << 'EOF'
[Unit]
Description=Stoatchat Media Proxy
After=network.target docker.service
Requires=docker.service

[Service]
Type=simple
User=root
WorkingDirectory=/root/stoatchat
ExecStart=/root/stoatchat/target/debug/revolt-january
Restart=always
RestartSec=10
StandardOutput=append:/root/stoatchat/proxy.log
StandardError=append:/root/stoatchat/proxy.log

[Install]
WantedBy=multi-user.target
EOF

# GIF Service
cat > /etc/systemd/system/stoatchat-gifbox.service << 'EOF'
[Unit]
Description=Stoatchat GIF Service
After=network.target docker.service
Requires=docker.service

[Service]
Type=simple
User=root
WorkingDirectory=/root/stoatchat
ExecStart=/root/stoatchat/target/debug/revolt-gifbox
Restart=always
RestartSec=10
StandardOutput=append:/root/stoatchat/gifbox.log
StandardError=append:/root/stoatchat/gifbox.log

[Install]
WantedBy=multi-user.target
EOF
```

### 2. Enable and Start Systemd Services
```bash
# Reload systemd
systemctl daemon-reload

# Enable services
systemctl enable stoatchat-api
systemctl enable stoatchat-events
systemctl enable stoatchat-files
systemctl enable stoatchat-proxy
systemctl enable stoatchat-gifbox

# Start services
systemctl start stoatchat-api
systemctl start stoatchat-events
systemctl start stoatchat-files
systemctl start stoatchat-proxy
systemctl start stoatchat-gifbox

# Check status
systemctl status stoatchat-api
systemctl status stoatchat-events
systemctl status stoatchat-files
systemctl status stoatchat-proxy
systemctl status stoatchat-gifbox
```

### 3. Manage with Systemd
```bash
# Start all services
systemctl start stoatchat-api stoatchat-events stoatchat-files stoatchat-proxy stoatchat-gifbox

# Stop all services
systemctl stop stoatchat-api stoatchat-events stoatchat-files stoatchat-proxy stoatchat-gifbox

# Restart all services
systemctl restart stoatchat-api stoatchat-events stoatchat-files stoatchat-proxy stoatchat-gifbox

# Check status of all services
systemctl status stoatchat-*
```

## Maintenance Tasks

### 1. Regular Maintenance
```bash
# Weekly maintenance script
cat > /root/stoatchat/weekly-maintenance.sh << 'EOF'
#!/bin/bash
cd /root/stoatchat

echo "=== Weekly Stoatchat Maintenance ==="
echo "Date: $(date)"

# Rotate logs
./rotate-logs.sh

# Update Docker images
docker-compose pull

# Restart services with new images
docker-compose down
docker-compose up -d

# Clean up old Docker images
docker image prune -f

# Check disk usage
echo "Disk usage:"
df -h /root/stoatchat

echo "Maintenance completed."
EOF

chmod +x /root/stoatchat/weekly-maintenance.sh
```

### 2. Update Procedures
```bash
# Update Stoatchat code
cd /root/stoatchat
git pull origin main

# Rebuild services
cargo build

# Restart services
./stop-all-services.sh
./start-all-services.sh
```

### 3. Backup Procedures
```bash
# Create backup script
cat > /root/stoatchat/backup.sh << 'EOF'
#!/bin/bash
BACKUP_DIR="/root/stoatchat-backups/$(date +%Y%m%d)"
mkdir -p "$BACKUP_DIR"

cd /root/stoatchat

# Backup configuration
cp Revolt.overrides.toml "$BACKUP_DIR/"
cp livekit.yml "$BACKUP_DIR/"
cp compose.yml "$BACKUP_DIR/"

# Backup MongoDB
docker exec stoatchat-mongodb mongodump --out "$BACKUP_DIR/mongodb"

# Backup MinIO data
docker exec stoatchat-minio tar czf - /data > "$BACKUP_DIR/minio-data.tar.gz"

echo "Backup completed: $BACKUP_DIR"
EOF

chmod +x /root/stoatchat/backup.sh
```

## Quick Reference

### Essential Commands
```bash
# Start everything
cd /root/stoatchat && ./start-all-services.sh

# Stop everything
cd /root/stoatchat && ./stop-all-services.sh

# Check status
ps aux | grep revolt && docker-compose ps

# View logs
cd /root/stoatchat && tail -f *.log

# Test endpoints
curl https://api.st.vish.gg/ && curl https://files.st.vish.gg/
```

### Service Ports
- API (revolt-delta): 14702
- Events (revolt-bonfire): 14703
- Files (revolt-autumn): 14704
- Proxy (revolt-january): 14705
- GIF (revolt-gifbox): 14706
- LiveKit: 7880
- Redis: 6380
- MongoDB: 27017
- MinIO: 14009

### Important Files
- Configuration: `/root/stoatchat/Revolt.overrides.toml`
- LiveKit config: `/root/stoatchat/livekit.yml`
- Docker config: `/root/stoatchat/compose.yml`
- Nginx config: `/etc/nginx/sites-available/stoatchat`
- Logs: `/root/stoatchat/*.log`