homelab-optimized/docs/admin/OPERATIONAL_STATUS.md

# Stoatchat Operational Status & Testing Documentation

## 🎯 Instance Overview
- **Domain**: st.vish.gg
- **Status**: ✅ **FULLY OPERATIONAL**
- **Deployment Date**: February 2026
- **Last Tested**: February 11, 2026
- **Platform**: Self-hosted Revolt chat server

## 🌐 Service Architecture

### Domain Structure
| Service | URL | Port | Status |
|---------|-----|------|--------|
| **Frontend** | https://st.vish.gg/ | 14702 | ✅ Active |
| **API** | https://api.st.vish.gg/ | 14702 | ✅ Active |
| **Events (WebSocket)** | wss://events.st.vish.gg/ | 14703 | ✅ Active |
| **Files** | https://files.st.vish.gg/ | 14704 | ✅ Active |
| **Proxy** | https://proxy.st.vish.gg/ | 14705 | ✅ Active |
| **Voice** | wss://voice.st.vish.gg/ | 7880 | ✅ Active |

### Infrastructure Components
- **Reverse Proxy**: Nginx with SSL termination
- **SSL Certificates**: Let's Encrypt (auto-renewal configured)
- **Database**: Redis (port 6380)
- **Voice/Video**: LiveKit integration
- **Email**: Gmail SMTP (your-email@example.com)

## 🧪 Comprehensive Testing Results

### Test Suite Summary
**Total Tests**: 6 categories  
**Passed**: 6/6 (100%)  
**Status**: ✅ **ALL TESTS PASSED**

### 1. Account Creation Test ✅
- **Method**: API POST to `/auth/account/create`
- **Test Email**: admin@example.com
- **Password**: REDACTED_PASSWORD
- **Result**: HTTP 204 (Success)
- **Account ID**: 01KH5RZXBHDX7W29XXFN6FB35F
- **Verification Token**: 2Kd_mgmImSvfNw2Mc8L1vi-oN0U0O5qL

### 2. Email Verification Test ✅
- **SMTP Server**: Gmail (smtp.gmail.com:587)
- **Sender**: your-email@example.com
- **Recipient**: admin@example.com
- **Delivery**: ✅ Successful
- **Verification**: ✅ Completed manually
- **Email System**: Fully functional

### 3. Authentication Test ✅
- **Login Method**: API POST to `/auth/session/login`
- **Credentials**: admin@example.com / REDACTED_PASSWORD
- **Result**: HTTP 200 (Success)
- **Session Token**: W_NfvzjWiukjVQEi30zNTmvPo4xo7pPJTKCZRvRP7TDQplfOjwgoad3AcuF9LEPI
- **Session ID**: 01KH5S1TG66V7BPZS8CFKHGSCR
- **User ID**: 01KH5RZXBHDX7W29XXFN6FB35F

### 4. Web Interface Test ✅
- **Frontend URL**: https://st.vish.gg/
- **Accessibility**: ✅ Fully accessible
- **Login Process**: ✅ Successful via web interface
- **UI Responsiveness**: ✅ Working correctly
- **SSL Certificate**: ✅ Valid and trusted

### 5. Real-time Messaging Test ✅
- **Test Channel**: Nerds channel
- **Message Sending**: ✅ Successful
- **Real-time Delivery**: ✅ Instant delivery
- **Channel Participation**: ✅ Full functionality
- **WebSocket Connection**: ✅ Stable

### 6. Infrastructure Health Test ✅
- **All Services**: ✅ Running and responsive
- **SSL Certificates**: ✅ Valid for all domains
- **DNS Resolution**: ✅ All subdomains resolving
- **Database Connection**: ✅ Redis connected
- **File Upload Service**: ✅ Operational
- **Voice/Video Service**: ✅ LiveKit integrated

## 📊 Performance Metrics

### Response Times
- **API Calls**: < 200ms average
- **Message Delivery**: < 1 second (real-time)
- **File Uploads**: Dependent on file size
- **Page Load**: < 2 seconds

### Uptime & Reliability
- **Target Uptime**: 99.9%
- **Current Status**: All services operational
- **Last Downtime**: None recorded
- **Monitoring**: Manual checks performed

## 🔐 Security Configuration

### SSL/TLS
- **Certificate Authority**: Let's Encrypt
- **Encryption**: TLS 1.2/1.3
- **HSTS**: Enabled
- **Certificate Renewal**: Automated

### Authentication
- **Method**: Session-based authentication
- **Password Requirements**: Enforced
- **Email Verification**: Required
- **Session Management**: Secure token-based

### Email Security
- **SMTP Authentication**: App-specific password
- **TLS Encryption**: Enabled
- **Authorized Recipients**: Limited to specific domains

## 📧 Email Configuration

### SMTP Settings
```toml
[api.smtp]
host = "smtp.gmail.com"
port = 587
username = "your-email@example.com"
password = "REDACTED_PASSWORD"
from_address = "your-email@example.com"
use_tls = true
```

### Authorized Email Recipients
- your-email@example.com
- admin@example.com
- user@example.com

## 🛠️ Service Management

### Starting Services
```bash
cd /root/stoatchat
./manage-services.sh start
```

### Checking Status
```bash
./manage-services.sh status
```

### Viewing Logs
```bash
# API logs
tail -f api.log

# Events logs
tail -f events.log

# Files logs
tail -f files.log

# Proxy logs
tail -f proxy.log
```

### Service Restart
```bash
./manage-services.sh restart
```

## 🔍 Monitoring & Maintenance

### Daily Checks
- [ ] Service status verification
- [ ] Log file review
- [ ] SSL certificate validity
- [ ] Disk space monitoring

### Weekly Checks
- [ ] Performance metrics review
- [ ] Security updates check
- [ ] Backup verification
- [ ] User activity monitoring

### Monthly Checks
- [ ] SSL certificate renewal
- [ ] System updates
- [ ] Configuration backup
- [ ] Performance optimization

## 🚨 Troubleshooting Guide

### Common Issues & Solutions

#### Services Not Starting
```bash
# Check logs for errors
tail -50 api.log

# Verify port availability
netstat -tulpn | grep :14702

# Restart specific service
./manage-services.sh restart
```

#### SSL Certificate Issues
```bash
# Check certificate status
openssl s_client -connect st.vish.gg:443 -servername st.vish.gg

# Renew certificates
sudo certbot renew

# Reload nginx
sudo systemctl reload nginx
```

#### Email Not Sending
1. Verify Gmail app password is valid
2. Check SMTP configuration in `Revolt.overrides.toml`
3. Test SMTP connection manually
4. Review API logs for email errors

#### Database Connection Issues
```bash
# Test Redis connection
redis-cli -p 6380 ping

# Check Redis status
sudo systemctl status redis-server

# Restart Redis if needed
sudo systemctl restart redis-server
```

## 📈 Usage Statistics

### Test Account Details
- **Email**: admin@example.com
- **Account ID**: 01KH5RZXBHDX7W29XXFN6FB35F
- **Status**: Verified and active
- **Last Login**: February 11, 2026
- **Test Messages**: Successfully sent in Nerds channel

### System Resources
- **CPU Usage**: Normal operation levels
- **Memory Usage**: Within expected parameters
- **Disk Space**: Adequate for current usage
- **Network**: All connections stable

## 🎯 Operational Readiness

### Production Readiness Checklist
- [x] All services deployed and running
- [x] SSL certificates installed and valid
- [x] Email system configured and tested
- [x] User registration working
- [x] Authentication system functional
- [x] Real-time messaging operational
- [x] File upload/download working
- [x] Voice/video calling available
- [x] Web interface accessible
- [x] API endpoints responding
- [x] Database connections stable
- [x] Monitoring procedures established

### Deployment Verification
- [x] Account creation tested
- [x] Email verification tested
- [x] Login process tested
- [x] Message sending tested
- [x] Channel functionality tested
- [x] Real-time features tested
- [x] SSL security verified
- [x] All domains accessible

## 📞 Support Information

### Technical Contacts
- **System Administrator**: your-email@example.com
- **Domain Owner**: vish.gg
- **Technical Support**: admin@example.com

### Emergency Procedures
1. **Service Outage**: Check service status and restart if needed
2. **SSL Issues**: Verify certificate validity and renew if necessary
3. **Database Problems**: Check Redis connection and restart service
4. **Email Issues**: Verify SMTP configuration and Gmail app password

### Escalation Path
1. Check service logs for error messages
2. Attempt service restart
3. Review configuration files
4. Contact system administrator if issues persist

## 🔄 Watchtower Auto-Update System

### System Overview
**Status**: ✅ **FULLY OPERATIONAL ACROSS ALL HOSTS**  
**Last Updated**: February 13, 2026  
**Configuration**: Scheduled updates with HTTP API monitoring

### Deployment Status by Host

| Host | Status | Schedule | Port | Network | Container ID |
|------|--------|----------|------|---------|--------------|
| **Homelab VM** | ✅ Running | 04:00 PST | 8083 | bridge | Active |
| **Calypso** | ✅ Running | 04:00 PST | 8080 | bridge | Active |
| **Atlantis** | ✅ Running | 02:00 PST | 8082 | prometheus-net | 51d8472bd7a4 |

### Configuration Features
- **Scheduled Updates**: Daily automatic container updates
- **Staggered Timing**: Prevents simultaneous updates across hosts
- **HTTP API**: Monitoring and metrics endpoints enabled
- **Prometheus Integration**: Metrics collection for monitoring
- **Dependency Management**: Rolling restart disabled where needed

### Monitoring Endpoints
```bash
# Homelab VM
curl -H "Authorization: Bearer REDACTED_WATCHTOWER_TOKEN" http://homelab-vm.local:8083/v1/update

# Calypso  
curl -H "Authorization: Bearer REDACTED_WATCHTOWER_TOKEN" http://calypso.local:8080/v1/update

# Atlantis
curl -H "Authorization: Bearer REDACTED_WATCHTOWER_TOKEN" http://atlantis.local:8082/v1/update
```

### Recent Fixes Applied
- **Port Conflicts**: Resolved by using unique ports per host
- **Dependency Issues**: Fixed rolling restart conflicts on Atlantis
- **Configuration Conflicts**: Removed polling/schedule conflicts on Calypso
- **Network Issues**: Created dedicated networks where needed

## 📝 Change Log

### February 13, 2026
- ✅ **Watchtower System Fully Operational**
- ✅ Fixed Atlantis dependency conflicts and port mapping
- ✅ Resolved Homelab VM port conflicts and notification URLs
- ✅ Fixed Calypso configuration conflicts
- ✅ All hosts now have scheduled auto-updates working
- ✅ HTTP API endpoints accessible for monitoring
- ✅ Comprehensive documentation created

### February 11, 2026
- ✅ Complete deployment testing performed
- ✅ All functionality verified operational
- ✅ Test account created and verified
- ✅ Real-time messaging confirmed working
- ✅ Documentation updated with test results

### Previous Changes
- Initial deployment completed
- SSL certificates configured
- Email system integrated
- All services deployed and configured

---

## 🎉 Final Status

**STOATCHAT INSTANCE STATUS: FULLY OPERATIONAL** ✅

The Stoatchat instance at st.vish.gg is completely functional and ready for production use. All core features have been tested and verified working, including:

- ✅ User registration and verification
- ✅ Authentication and session management  
- ✅ Real-time messaging and channels
- ✅ File sharing capabilities
- ✅ Voice/video calling integration
- ✅ Web interface accessibility
- ✅ API functionality
- ✅ Email notifications
- ✅ SSL security

**The deployment is complete and the service is ready for end users.**

---

**Document Version**: 1.0  
**Last Updated**: February 11, 2026  
**Next Review**: February 18, 2026