# GitOps Deployment Comprehensive Guide

*Last Updated: February 14, 2026*

## 🎯 Overview

This homelab infrastructure is deployed using **GitOps methodology** with **Portainer Enterprise Edition** as the orchestration platform. All services are defined as Docker Compose files in this Git repository and automatically deployed across multiple hosts.

## 🏗️ GitOps Architecture

### Core Components
- **Git Repository**: Source of truth for all infrastructure configurations
- **Portainer EE**: GitOps orchestration and container management (v2.33.7)
- **Docker Compose**: Service definition and deployment format
- **Multi-Host Deployment**: Services distributed across Synology NAS, VMs, and edge devices

### Current Deployment Status
**Verified Active Stacks**: 18 compose stacks deployed via GitOps
**Total Containers**: 50+ containers across infrastructure
**Management Interface**: https://192.168.0.200:9443 (Portainer EE)

## 📊 Active GitOps Deployments

### Atlantis (Primary NAS) - 18 Stacks Active

| Stack Name | Containers | Config Path | Status |
|------------|------------|-------------|---------|
| **arr-stack** | 18 | `Atlantis/arr-suite/arrs-compose.yaml` | ✅ Running |
| **baikal-stack** | 1 | `Atlantis/baikal/baikal.yaml` | ✅ Running |
| **dokuwiki-stack** | 1 | `Atlantis/dokuwiki.yml` | ✅ Running |
| **dyndns-updater-stack** | 3 | `Atlantis/dynamicdnsupdater.yaml` | ✅ Running |
| **fenrus-stack** | 1 | `Atlantis/fenrus.yaml` | ✅ Running |
| **homarr-stack** | 1 | `hosts/synology/atlantis/homarr.yaml` | ✅ Running |
| **immich-stack** | 4 | `Atlantis/immich/docker-compose.yml` | ✅ Running |
| **iperf3-stack** | 1 | `Atlantis/iperf3.yaml` | ✅ Running |
| **it_tools-stack** | 1 | `Atlantis/it_tools.yml` | ✅ Running |
| **jitsi** | 5 | `Atlantis/jitsi/jitsi.yml` | ✅ Running |
| **joplin-stack** | 2 | `Atlantis/joplin.yml` | ✅ Running |
| **node-exporter-stack** | 2 | `Atlantis/grafana_prometheus/atlantis_node_exporter.yaml` | ✅ Running |
| **ollama** | 2 | `Atlantis/ollama/docker-compose.yml` | ✅ Running |
| **syncthing-stack** | 1 | `Atlantis/syncthing.yml` | ✅ Running |
| **theme-park** | 1 | `Atlantis/theme-park/theme-park.yaml` | ✅ Running |
| **vaultwarden-stack** | 2 | `Atlantis/vaultwarden.yaml` | ✅ Running |
| **watchtower-stack** | 1 | `common/watchtower-full.yaml` | ✅ Running |
| **youtubedl** | 1 | `Atlantis/youtubedl.yaml` | ✅ Running |

### Other Hosts
- **Calypso**: Synology DS723+ (Secondary NAS)
- **Homelab VM**: Proxmox VM (Cloud Services)
- **Concord NUC**: Intel NUC (Edge Computing)
- **Raspberry Pi 5**: ARM64 (IoT/Edge)

## 🚀 GitOps Workflow

### 1. Service Definition
Services are defined using Docker Compose YAML files in the repository:

```yaml
# Example: Atlantis/new-service.yaml
version: '3.8'
services:
  new-service:
    image: example/service:latest
    container_name: new-service
    ports:
      - "8080:8080"
    environment:
      - ENV_VAR=value
    volumes:
      - /volume1/docker/new-service:/data
    restart: unless-stopped
```

### 2. Git Commit & Push
```bash
# Add new service configuration
git add Atlantis/new-service.yaml
git commit -m "Add new service deployment

- Configure new-service with proper volumes
- Set up environment variables
- Enable auto-restart policy"

# Push to trigger GitOps deployment
git push origin main
```

### 3. Automatic Deployment
- Portainer monitors the Git repository for changes
- New commits trigger automatic stack updates
- Services are deployed/updated across the infrastructure
- Health checks verify successful deployment

### 4. Monitoring & Verification
```bash
# Check deployment status
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker compose ls"

# Verify service health
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker ps | grep new-service"
```

## 📁 Repository Structure for GitOps

### Host-Specific Configurations
```
homelab/
├── Atlantis/                    # Synology DS1823xs+ (Primary NAS)
│   ├── arr-suite/              # Media automation stack
│   ├── immich/                 # Photo management
│   ├── ollama/                 # AI/LLM services
│   └── *.yaml                  # Individual service configs
├── Calypso/                    # Synology DS723+ (Secondary NAS)
│   ├── grafana_prometheus/     # Monitoring stack
│   ├── immich/                 # Photo backup
│   └── *.yaml                  # Service configurations
├── homelab_vm/                 # Proxmox VM (Cloud Services)
│   ├── monitoring.yaml         # Production monitoring
│   ├── stoatchat.yaml         # Chat platform
│   └── *.yaml                  # Cloud service configs
├── concord_nuc/                # Intel NUC (Edge Computing)
│   ├── homeassistant.yaml     # Home automation
│   ├── plex.yaml              # Media server
│   └── *.yaml                  # Edge service configs
└── common/                     # Shared configurations
    └── watchtower-*.yaml       # Auto-update services
```

### Service Categories
- **Media & Entertainment**: Plex, Jellyfin, *arr suite, Immich
- **Development & DevOps**: Gitea, Portainer, monitoring stack
- **Productivity**: PaperlessNGX, Joplin, Syncthing
- **Network & Infrastructure**: AdGuard, Nginx Proxy Manager, Authentik
- **Communication**: Stoatchat, Matrix, Jitsi
- **Utilities**: Watchtower, theme-park, IT Tools

## 🔧 Service Management Operations

### Adding a New Service

1. **Create Service Configuration**
```bash
# Create new service file
cat > Atlantis/new-service.yaml << 'EOF'
version: '3.8'
services:
  new-service:
    image: example/service:latest
    container_name: new-service
    ports:
      - "8080:8080"
    volumes:
      - /volume1/docker/new-service:/data
    restart: unless-stopped
EOF
```

2. **Commit and Deploy**
```bash
git add Atlantis/new-service.yaml
git commit -m "Add new-service deployment"
git push origin main
```

3. **Verify Deployment**
```bash
# Check if stack was created
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker compose ls | grep new-service"

# Verify container is running
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker ps | grep new-service"
```

### Updating an Existing Service

1. **Modify Configuration**
```bash
# Edit existing service
nano Atlantis/existing-service.yaml
```

2. **Commit Changes**
```bash
git add Atlantis/existing-service.yaml
git commit -m "Update existing-service configuration

- Upgrade to latest image version
- Add new environment variables
- Update volume mounts"
git push origin main
```

3. **Monitor Update**
- Portainer will automatically pull changes
- Service will be redeployed with new configuration
- Check Portainer UI for deployment status

### Removing a Service

1. **Remove Configuration File**
```bash
git rm Atlantis/old-service.yaml
git commit -m "Remove old-service deployment"
git push origin main
```

2. **Manual Cleanup (if needed)**
```bash
# Remove any persistent volumes or data
ssh -p 60000 vish@192.168.0.200 "sudo rm -rf /volume1/docker/old-service"
```

## 🔍 Monitoring & Troubleshooting

### GitOps Health Checks

#### Check Portainer Status
```bash
# Verify Portainer is running
curl -k -s "https://192.168.0.200:9443/api/system/status"

# Check container status
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker ps | grep portainer"
```

#### Verify Git Sync Status
```bash
# Check if Portainer can access Git repository
# (Check via Portainer UI: Stacks → Repository sync status)

# Verify latest commits are reflected
git log --oneline -5
```

#### Monitor Stack Deployments
```bash
# List all active stacks
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker compose ls"

# Check specific stack status
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker compose -f /path/to/stack.yaml ps"
```

### Common Issues & Solutions

#### Stack Deployment Fails
1. **Check YAML Syntax**
```bash
# Validate YAML syntax
yamllint Atlantis/service.yaml

# Check Docker Compose syntax
docker-compose -f Atlantis/service.yaml config
```

2. **Review Portainer Logs**
```bash
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker logs portainer"
```

3. **Check Resource Constraints**
```bash
# Verify disk space
ssh -p 60000 vish@192.168.0.200 "df -h"

# Check memory usage
ssh -p 60000 vish@192.168.0.200 "free -h"
```

#### Git Repository Access Issues
1. **Verify Repository URL**
2. **Check Authentication credentials**
3. **Confirm network connectivity**

#### Service Won't Start
1. **Check container logs**
```bash
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker logs service-name"
```

2. **Verify port conflicts**
```bash
ssh -p 60000 vish@192.168.0.200 "sudo netstat -tulpn | grep :PORT"
```

3. **Check volume mounts**
```bash
ssh -p 60000 vish@192.168.0.200 "ls -la /volume1/docker/service-name"
```

## 🔐 Security Considerations

### GitOps Security Best Practices
- **Repository Access**: Secure Git repository with appropriate access controls
- **Secrets Management**: Use Docker secrets or external secret management
- **Network Security**: Services deployed on isolated Docker networks
- **Regular Updates**: Watchtower ensures containers stay updated

### Access Control
- **Portainer Authentication**: Multi-user access with role-based permissions
- **SSH Access**: Key-based authentication for server management
- **Service Authentication**: Individual service authentication where applicable

## 📈 Performance & Scaling

### Resource Monitoring
- **Container Metrics**: Monitor CPU, memory, and disk usage
- **Network Performance**: Track bandwidth and connection metrics
- **Storage Utilization**: Monitor disk space across all hosts

### Scaling Strategies
- **Horizontal Scaling**: Deploy services across multiple hosts
- **Load Balancing**: Use Nginx Proxy Manager for traffic distribution
- **Resource Optimization**: Optimize container resource limits

## 🔄 Backup & Disaster Recovery

### GitOps Backup Strategy
- **Repository Backup**: Git repository is the source of truth
- **Configuration Backup**: All service configurations version controlled
- **Data Backup**: Persistent volumes backed up separately

### Recovery Procedures
1. **Service Recovery**: Redeploy from Git repository
2. **Data Recovery**: Restore from backup volumes
3. **Full Infrastructure Recovery**: Bootstrap new hosts with GitOps

## 📚 Related Documentation

- [GITOPS_DEPLOYMENT_GUIDE.md](../GITOPS_DEPLOYMENT_GUIDE.md) - Original deployment guide
- [MONITORING_ARCHITECTURE.md](../MONITORING_ARCHITECTURE.md) - Monitoring setup
- [docs/admin/portainer-backup.md](portainer-backup.md) - Portainer backup procedures
- [docs/runbooks/add-new-service.md](../runbooks/add-new-service.md) - Service deployment runbook

## 🎯 Next Steps

### Short Term
- [ ] Set up automated GitOps health monitoring
- [ ] Create service deployment templates
- [ ] Implement automated testing for configurations

### Medium Term
- [ ] Expand GitOps to additional hosts
- [ ] Implement blue-green deployments
- [ ] Add configuration validation pipelines

### Long Term
- [ ] Migrate to Kubernetes GitOps (ArgoCD/Flux)
- [ ] Implement infrastructure as code (Terraform)
- [ ] Add automated disaster recovery testing

---

**Document Status**: ✅ Active  
**Deployment Method**: GitOps via Portainer EE  
**Last Verified**: February 14, 2026  
**Next Review**: March 14, 2026