357 lines
11 KiB
Markdown
357 lines
11 KiB
Markdown
# GitOps Deployment Comprehensive Guide
|
|
|
|
*Last Updated: February 14, 2026*
|
|
|
|
## 🎯 Overview
|
|
|
|
This homelab infrastructure is deployed using **GitOps methodology** with **Portainer Enterprise Edition** as the orchestration platform. All services are defined as Docker Compose files in this Git repository and automatically deployed across multiple hosts.
|
|
|
|
## 🏗️ GitOps Architecture
|
|
|
|
### Core Components
|
|
- **Git Repository**: Source of truth for all infrastructure configurations
|
|
- **Portainer EE**: GitOps orchestration and container management (v2.33.7)
|
|
- **Docker Compose**: Service definition and deployment format
|
|
- **Multi-Host Deployment**: Services distributed across Synology NAS, VMs, and edge devices
|
|
|
|
### Current Deployment Status
|
|
**Verified Active Stacks**: 18 compose stacks deployed via GitOps
|
|
**Total Containers**: 50+ containers across infrastructure
|
|
**Management Interface**: https://192.168.0.200:9443 (Portainer EE)
|
|
|
|
## 📊 Active GitOps Deployments
|
|
|
|
### Atlantis (Primary NAS) - 18 Stacks Active
|
|
|
|
| Stack Name | Containers | Config Path | Status |
|
|
|------------|------------|-------------|---------|
|
|
| **arr-stack** | 18 | `Atlantis/arr-suite/arrs-compose.yaml` | ✅ Running |
|
|
| **baikal-stack** | 1 | `Atlantis/baikal/baikal.yaml` | ✅ Running |
|
|
| **dokuwiki-stack** | 1 | `Atlantis/dokuwiki.yml` | ✅ Running |
|
|
| **dyndns-updater-stack** | 3 | `Atlantis/dynamicdnsupdater.yaml` | ✅ Running |
|
|
| **fenrus-stack** | 1 | `Atlantis/fenrus.yaml` | ✅ Running |
|
|
| **homarr-stack** | 1 | `hosts/synology/atlantis/homarr.yaml` | ✅ Running |
|
|
| **immich-stack** | 4 | `Atlantis/immich/docker-compose.yml` | ✅ Running |
|
|
| **iperf3-stack** | 1 | `Atlantis/iperf3.yaml` | ✅ Running |
|
|
| **it_tools-stack** | 1 | `Atlantis/it_tools.yml` | ✅ Running |
|
|
| **jitsi** | 5 | `Atlantis/jitsi/jitsi.yml` | ✅ Running |
|
|
| **joplin-stack** | 2 | `Atlantis/joplin.yml` | ✅ Running |
|
|
| **node-exporter-stack** | 2 | `Atlantis/grafana_prometheus/atlantis_node_exporter.yaml` | ✅ Running |
|
|
| **ollama** | 2 | `Atlantis/ollama/docker-compose.yml` | ✅ Running |
|
|
| **syncthing-stack** | 1 | `Atlantis/syncthing.yml` | ✅ Running |
|
|
| **theme-park** | 1 | `Atlantis/theme-park/theme-park.yaml` | ✅ Running |
|
|
| **vaultwarden-stack** | 2 | `Atlantis/vaultwarden.yaml` | ✅ Running |
|
|
| **watchtower-stack** | 1 | `common/watchtower-full.yaml` | ✅ Running |
|
|
| **youtubedl** | 1 | `Atlantis/youtubedl.yaml` | ✅ Running |
|
|
|
|
### Other Hosts
|
|
- **Calypso**: Synology DS723+ (Secondary NAS)
|
|
- **Homelab VM**: Proxmox VM (Cloud Services)
|
|
- **Concord NUC**: Intel NUC (Edge Computing)
|
|
- **Raspberry Pi 5**: ARM64 (IoT/Edge)
|
|
|
|
## 🚀 GitOps Workflow
|
|
|
|
### 1. Service Definition
|
|
Services are defined using Docker Compose YAML files in the repository:
|
|
|
|
```yaml
|
|
# Example: Atlantis/new-service.yaml
|
|
version: '3.8'
|
|
services:
|
|
new-service:
|
|
image: example/service:latest
|
|
container_name: new-service
|
|
ports:
|
|
- "8080:8080"
|
|
environment:
|
|
- ENV_VAR=value
|
|
volumes:
|
|
- /volume1/docker/new-service:/data
|
|
restart: unless-stopped
|
|
```
|
|
|
|
### 2. Git Commit & Push
|
|
```bash
|
|
# Add new service configuration
|
|
git add Atlantis/new-service.yaml
|
|
git commit -m "Add new service deployment
|
|
|
|
- Configure new-service with proper volumes
|
|
- Set up environment variables
|
|
- Enable auto-restart policy"
|
|
|
|
# Push to trigger GitOps deployment
|
|
git push origin main
|
|
```
|
|
|
|
### 3. Automatic Deployment
|
|
- Portainer monitors the Git repository for changes
|
|
- New commits trigger automatic stack updates
|
|
- Services are deployed/updated across the infrastructure
|
|
- Health checks verify successful deployment
|
|
|
|
### 4. Monitoring & Verification
|
|
```bash
|
|
# Check deployment status
|
|
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker compose ls"
|
|
|
|
# Verify service health
|
|
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker ps | grep new-service"
|
|
```
|
|
|
|
## 📁 Repository Structure for GitOps
|
|
|
|
### Host-Specific Configurations
|
|
```
|
|
homelab/
|
|
├── Atlantis/ # Synology DS1823xs+ (Primary NAS)
|
|
│ ├── arr-suite/ # Media automation stack
|
|
│ ├── immich/ # Photo management
|
|
│ ├── ollama/ # AI/LLM services
|
|
│ └── *.yaml # Individual service configs
|
|
├── Calypso/ # Synology DS723+ (Secondary NAS)
|
|
│ ├── grafana_prometheus/ # Monitoring stack
|
|
│ ├── immich/ # Photo backup
|
|
│ └── *.yaml # Service configurations
|
|
├── homelab_vm/ # Proxmox VM (Cloud Services)
|
|
│ ├── monitoring.yaml # Production monitoring
|
|
│ ├── stoatchat.yaml # Chat platform
|
|
│ └── *.yaml # Cloud service configs
|
|
├── concord_nuc/ # Intel NUC (Edge Computing)
|
|
│ ├── homeassistant.yaml # Home automation
|
|
│ ├── plex.yaml # Media server
|
|
│ └── *.yaml # Edge service configs
|
|
└── common/ # Shared configurations
|
|
└── watchtower-*.yaml # Auto-update services
|
|
```
|
|
|
|
### Service Categories
|
|
- **Media & Entertainment**: Plex, Jellyfin, *arr suite, Immich
|
|
- **Development & DevOps**: Gitea, Portainer, monitoring stack
|
|
- **Productivity**: PaperlessNGX, Joplin, Syncthing
|
|
- **Network & Infrastructure**: AdGuard, Nginx Proxy Manager, Authentik
|
|
- **Communication**: Stoatchat, Matrix, Jitsi
|
|
- **Utilities**: Watchtower, theme-park, IT Tools
|
|
|
|
## 🔧 Service Management Operations
|
|
|
|
### Adding a New Service
|
|
|
|
1. **Create Service Configuration**
|
|
```bash
|
|
# Create new service file
|
|
cat > Atlantis/new-service.yaml << 'EOF'
|
|
version: '3.8'
|
|
services:
|
|
new-service:
|
|
image: example/service:latest
|
|
container_name: new-service
|
|
ports:
|
|
- "8080:8080"
|
|
volumes:
|
|
- /volume1/docker/new-service:/data
|
|
restart: unless-stopped
|
|
EOF
|
|
```
|
|
|
|
2. **Commit and Deploy**
|
|
```bash
|
|
git add Atlantis/new-service.yaml
|
|
git commit -m "Add new-service deployment"
|
|
git push origin main
|
|
```
|
|
|
|
3. **Verify Deployment**
|
|
```bash
|
|
# Check if stack was created
|
|
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker compose ls | grep new-service"
|
|
|
|
# Verify container is running
|
|
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker ps | grep new-service"
|
|
```
|
|
|
|
### Updating an Existing Service
|
|
|
|
1. **Modify Configuration**
|
|
```bash
|
|
# Edit existing service
|
|
nano Atlantis/existing-service.yaml
|
|
```
|
|
|
|
2. **Commit Changes**
|
|
```bash
|
|
git add Atlantis/existing-service.yaml
|
|
git commit -m "Update existing-service configuration
|
|
|
|
- Upgrade to latest image version
|
|
- Add new environment variables
|
|
- Update volume mounts"
|
|
git push origin main
|
|
```
|
|
|
|
3. **Monitor Update**
|
|
- Portainer will automatically pull changes
|
|
- Service will be redeployed with new configuration
|
|
- Check Portainer UI for deployment status
|
|
|
|
### Removing a Service
|
|
|
|
1. **Remove Configuration File**
|
|
```bash
|
|
git rm Atlantis/old-service.yaml
|
|
git commit -m "Remove old-service deployment"
|
|
git push origin main
|
|
```
|
|
|
|
2. **Manual Cleanup (if needed)**
|
|
```bash
|
|
# Remove any persistent volumes or data
|
|
ssh -p 60000 vish@192.168.0.200 "sudo rm -rf /volume1/docker/old-service"
|
|
```
|
|
|
|
## 🔍 Monitoring & Troubleshooting
|
|
|
|
### GitOps Health Checks
|
|
|
|
#### Check Portainer Status
|
|
```bash
|
|
# Verify Portainer is running
|
|
curl -k -s "https://192.168.0.200:9443/api/system/status"
|
|
|
|
# Check container status
|
|
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker ps | grep portainer"
|
|
```
|
|
|
|
#### Verify Git Sync Status
|
|
```bash
|
|
# Check if Portainer can access Git repository
|
|
# (Check via Portainer UI: Stacks → Repository sync status)
|
|
|
|
# Verify latest commits are reflected
|
|
git log --oneline -5
|
|
```
|
|
|
|
#### Monitor Stack Deployments
|
|
```bash
|
|
# List all active stacks
|
|
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker compose ls"
|
|
|
|
# Check specific stack status
|
|
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker compose -f /path/to/stack.yaml ps"
|
|
```
|
|
|
|
### Common Issues & Solutions
|
|
|
|
#### Stack Deployment Fails
|
|
1. **Check YAML Syntax**
|
|
```bash
|
|
# Validate YAML syntax
|
|
yamllint Atlantis/service.yaml
|
|
|
|
# Check Docker Compose syntax
|
|
docker-compose -f Atlantis/service.yaml config
|
|
```
|
|
|
|
2. **Review Portainer Logs**
|
|
```bash
|
|
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker logs portainer"
|
|
```
|
|
|
|
3. **Check Resource Constraints**
|
|
```bash
|
|
# Verify disk space
|
|
ssh -p 60000 vish@192.168.0.200 "df -h"
|
|
|
|
# Check memory usage
|
|
ssh -p 60000 vish@192.168.0.200 "free -h"
|
|
```
|
|
|
|
#### Git Repository Access Issues
|
|
1. **Verify Repository URL**
|
|
2. **Check Authentication credentials**
|
|
3. **Confirm network connectivity**
|
|
|
|
#### Service Won't Start
|
|
1. **Check container logs**
|
|
```bash
|
|
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker logs service-name"
|
|
```
|
|
|
|
2. **Verify port conflicts**
|
|
```bash
|
|
ssh -p 60000 vish@192.168.0.200 "sudo netstat -tulpn | grep :PORT"
|
|
```
|
|
|
|
3. **Check volume mounts**
|
|
```bash
|
|
ssh -p 60000 vish@192.168.0.200 "ls -la /volume1/docker/service-name"
|
|
```
|
|
|
|
## 🔐 Security Considerations
|
|
|
|
### GitOps Security Best Practices
|
|
- **Repository Access**: Secure Git repository with appropriate access controls
|
|
- **Secrets Management**: Use Docker secrets or external secret management
|
|
- **Network Security**: Services deployed on isolated Docker networks
|
|
- **Regular Updates**: Watchtower ensures containers stay updated
|
|
|
|
### Access Control
|
|
- **Portainer Authentication**: Multi-user access with role-based permissions
|
|
- **SSH Access**: Key-based authentication for server management
|
|
- **Service Authentication**: Individual service authentication where applicable
|
|
|
|
## 📈 Performance & Scaling
|
|
|
|
### Resource Monitoring
|
|
- **Container Metrics**: Monitor CPU, memory, and disk usage
|
|
- **Network Performance**: Track bandwidth and connection metrics
|
|
- **Storage Utilization**: Monitor disk space across all hosts
|
|
|
|
### Scaling Strategies
|
|
- **Horizontal Scaling**: Deploy services across multiple hosts
|
|
- **Load Balancing**: Use Nginx Proxy Manager for traffic distribution
|
|
- **Resource Optimization**: Optimize container resource limits
|
|
|
|
## 🔄 Backup & Disaster Recovery
|
|
|
|
### GitOps Backup Strategy
|
|
- **Repository Backup**: Git repository is the source of truth
|
|
- **Configuration Backup**: All service configurations version controlled
|
|
- **Data Backup**: Persistent volumes backed up separately
|
|
|
|
### Recovery Procedures
|
|
1. **Service Recovery**: Redeploy from Git repository
|
|
2. **Data Recovery**: Restore from backup volumes
|
|
3. **Full Infrastructure Recovery**: Bootstrap new hosts with GitOps
|
|
|
|
## 📚 Related Documentation
|
|
|
|
- [GITOPS_DEPLOYMENT_GUIDE.md](../GITOPS_DEPLOYMENT_GUIDE.md) - Original deployment guide
|
|
- [MONITORING_ARCHITECTURE.md](../MONITORING_ARCHITECTURE.md) - Monitoring setup
|
|
- [docs/admin/portainer-backup.md](portainer-backup.md) - Portainer backup procedures
|
|
- [docs/runbooks/add-new-service.md](../runbooks/add-new-service.md) - Service deployment runbook
|
|
|
|
## 🎯 Next Steps
|
|
|
|
### Short Term
|
|
- [ ] Set up automated GitOps health monitoring
|
|
- [ ] Create service deployment templates
|
|
- [ ] Implement automated testing for configurations
|
|
|
|
### Medium Term
|
|
- [ ] Expand GitOps to additional hosts
|
|
- [ ] Implement blue-green deployments
|
|
- [ ] Add configuration validation pipelines
|
|
|
|
### Long Term
|
|
- [ ] Migrate to Kubernetes GitOps (ArgoCD/Flux)
|
|
- [ ] Implement infrastructure as code (Terraform)
|
|
- [ ] Add automated disaster recovery testing
|
|
|
|
---
|
|
|
|
**Document Status**: ✅ Active
|
|
**Deployment Method**: GitOps via Portainer EE
|
|
**Last Verified**: February 14, 2026
|
|
**Next Review**: March 14, 2026 |