# GitOps Deployment Comprehensive Guide

*Last Updated: March 8, 2026*

## 🎯 Overview

This homelab infrastructure is deployed using **GitOps methodology** with **Portainer Enterprise Edition** as the orchestration platform. All services are defined as Docker Compose files in this Git repository and automatically deployed across multiple hosts.

## 🏗️ GitOps Architecture

### Core Components
- **Git Repository**: Source of truth for all infrastructure configurations
- **Portainer EE**: GitOps orchestration and container management (v2.33.7)
- **Docker Compose**: Service definition and deployment format
- **Multi-Host Deployment**: Services distributed across Synology NAS, VMs, and edge devices

### Current Deployment Status
**Verified Active Stacks**: 81 compose stacks across 5 endpoints — all GitOps-managed
**Total Containers**: 157+ containers across infrastructure
**Management Interface**: https://192.168.0.200:9443 (Portainer EE)

## 📊 Active GitOps Deployments

All 5 endpoints are fully GitOps-managed. Every stack uses the canonical `hosts/` path.

### Atlantis (Primary NAS, ep=2) — 24 Stacks

| Stack Name | Config Path | Status |
|------------|-------------|--------|
| **arr-stack** | `hosts/synology/atlantis/arr-suite/arrs-compose.yaml` | ✅ Running |
| **audiobookshelf-stack** | `hosts/synology/atlantis/audiobookshelf.yaml` | ✅ Running |
| **baikal-stack** | `hosts/synology/atlantis/baikal/baikal.yaml` | ✅ Running |
| **calibre-stack** | `hosts/synology/atlantis/calibre.yaml` | ⏸ Stopped (intentional) |
| **dokuwiki-stack** | `hosts/synology/atlantis/dokuwiki.yml` | ✅ Running |
| **dyndns-updater-stack** | `hosts/synology/atlantis/dynamicdnsupdater.yaml` | ✅ Running |
| **fenrus-stack** | `hosts/synology/atlantis/fenrus.yaml` | ✅ Running |
| **homarr-stack** | `hosts/synology/atlantis/homarr.yaml` | ✅ Running |
| **immich-stack** | `hosts/synology/atlantis/immich/docker-compose.yml` | ✅ Running |
| **iperf3-stack** | `hosts/synology/atlantis/iperf3.yaml` | ✅ Running |
| **it_tools-stack** | `hosts/synology/atlantis/it_tools.yml` | ✅ Running |
| **jitsi-stack** | `hosts/synology/atlantis/jitsi/jitsi.yml` | ✅ Running |
| **joplin-stack** | `hosts/synology/atlantis/joplin.yml` | ✅ Running |
| **node-exporter-stack** | `hosts/synology/atlantis/grafana_prometheus/atlantis_node_exporter.yaml` | ✅ Running |
| **ollama-stack** | `hosts/synology/atlantis/ollama/docker-compose.yml` | ⏸ Stopped (intentional) |
| **syncthing-stack** | `hosts/synology/atlantis/syncthing.yml` | ✅ Running |
| **theme-park-stack** | `hosts/synology/atlantis/theme-park/theme-park.yaml` | ✅ Running |
| **vaultwarden-stack** | `hosts/synology/atlantis/vaultwarden.yaml` | ✅ Running |
| **watchtower-stack** | `common/watchtower-full.yaml` | ✅ Running |
| **youtubedl-stack** | `hosts/synology/atlantis/youtubedl.yaml` | ✅ Running |

### Calypso (Secondary NAS, ep=443397) — 23 Stacks

22 managed stacks fully GitOps; `gitea` (id=249) intentionally kept as manual (bootstrap dependency).

| Stack Name | Config Path | Status |
|------------|-------------|--------|
| **actual-budget-stack** | `hosts/synology/calypso/actualbudget.yml` | ✅ Running |
| **adguard-stack** | `hosts/synology/calypso/adguard.yaml` | ✅ Running |
| **apt-cacher-ng-stack** | `hosts/synology/calypso/apt-cacher-ng/apt-cacher-ng.yml` | ✅ Running |
| **arr-stack** | `hosts/synology/calypso/arr_suite_with_dracula.yml` | ✅ Running |
| **authentik-sso-stack** | `hosts/synology/calypso/authentik/docker-compose.yaml` | ✅ Running |
| **diun-stack** | `hosts/synology/calypso/diun.yaml` | ✅ Running |
| **dozzle-agent-stack** | `hosts/synology/calypso/dozzle-agent.yaml` | ✅ Running |
| **gitea** (manual) | — | ✅ Running |
| **gitea-runner-stack** | `hosts/synology/calypso/gitea-runner.yaml` | ✅ Running |
| **immich-stack** | `hosts/synology/calypso/immich/docker-compose.yml` | ✅ Running |
| **iperf3-stack** | `hosts/synology/calypso/iperf3.yml` | ✅ Running |
| **node-exporter-stack** | `hosts/synology/calypso/node-exporter.yaml` | ✅ Running |
| **openspeedtest-stack** | `hosts/synology/calypso/openspeedtest.yaml` | ✅ Running |
| **paperless-ai-stack** | `hosts/synology/calypso/paperless/paperless-ai.yml` | ✅ Running |
| **paperless-stack** | `hosts/synology/calypso/paperless/docker-compose.yml` | ✅ Running |
| **rackula-stack** | `hosts/synology/calypso/rackula.yml` | ✅ Running |
| **retro-site-stack** | `hosts/synology/calypso/retro-site.yaml` | ✅ Running |
| **rustdesk-stack** | `hosts/synology/calypso/rustdesk.yaml` | ✅ Running |
| **scrutiny-collector-stack** | `hosts/synology/calypso/scrutiny-collector.yaml` | ✅ Running |
| **seafile-new-stack** | `hosts/synology/calypso/seafile-new.yaml` | ✅ Running |
| **syncthing-stack** | `hosts/synology/calypso/syncthing.yaml` | ✅ Running |
| **watchtower-stack** | `common/watchtower-full.yaml` | ✅ Running |
| **wireguard-stack** | `hosts/synology/calypso/wireguard-server.yaml` | ✅ Running |

### Concord NUC (ep=443398) — 11 Stacks

| Stack Name | Config Path | Status |
|------------|-------------|--------|
| **adguard-stack** | `hosts/physical/concord-nuc/adguard.yaml` | ✅ Running |
| **diun-stack** | `hosts/physical/concord-nuc/diun.yaml` | ✅ Running |
| **dozzle-agent-stack** | `hosts/physical/concord-nuc/dozzle-agent.yaml` | ✅ Running |
| **dyndns-updater-stack** | `hosts/physical/concord-nuc/dyndns_updater.yaml` | ✅ Running |
| **homeassistant-stack** | `hosts/physical/concord-nuc/homeassistant.yaml` | ✅ Running |
| **invidious-stack** | `hosts/physical/concord-nuc/invidious/invidious.yaml` | ✅ Running |
| **plex-stack** | `hosts/physical/concord-nuc/plex.yaml` | ✅ Running |
| **scrutiny-collector-stack** | `hosts/physical/concord-nuc/scrutiny-collector.yaml` | ✅ Running |
| **syncthing-stack** | `hosts/physical/concord-nuc/syncthing.yaml` | ✅ Running |
| **wireguard-stack** | `hosts/physical/concord-nuc/wireguard.yaml` | ✅ Running |
| **yourspotify-stack** | `hosts/physical/concord-nuc/yourspotify.yaml` | ✅ Running |

### Homelab VM (ep=443399) — 19 Stacks

| Stack Name | Config Path | Status |
|------------|-------------|--------|
| **alerting-stack** | `hosts/vms/homelab-vm/alerting.yaml` | ✅ Running |
| **archivebox-stack** | `hosts/vms/homelab-vm/archivebox.yaml` | ✅ Running |
| **binternet-stack** | `hosts/vms/homelab-vm/binternet.yaml` | ✅ Running |
| **diun-stack** | `hosts/vms/homelab-vm/diun.yaml` | ✅ Running |
| **dozzle-agent-stack** | `hosts/vms/homelab-vm/dozzle-agent.yaml` | ✅ Running |
| **drawio-stack** | `hosts/vms/homelab-vm/drawio.yml` | ✅ Running |
| **hoarder-karakeep-stack** | `hosts/vms/homelab-vm/hoarder.yaml` | ✅ Running |
| **monitoring-stack** | `hosts/vms/homelab-vm/monitoring.yaml` | ✅ Running |
| **ntfy-stack** | `hosts/vms/homelab-vm/ntfy.yaml` | ✅ Running |
| **openhands-stack** | `hosts/vms/homelab-vm/openhands.yaml` | ✅ Running |
| **perplexica-stack** | `hosts/vms/homelab-vm/perplexica.yaml` | ✅ Running |
| **proxitok-stack** | `hosts/vms/homelab-vm/proxitok.yaml` | ✅ Running |
| **redlib-stack** | `hosts/vms/homelab-vm/redlib.yaml` | ✅ Running |
| **scrutiny-stack** | `hosts/vms/homelab-vm/scrutiny.yaml` | ✅ Running |
| **signal-api-stack** | `hosts/vms/homelab-vm/signal_api.yaml` | ✅ Running |
| **syncthing-stack** | `hosts/vms/homelab-vm/syncthing.yml` | ✅ Running |
| **watchyourlan-stack** | `hosts/vms/homelab-vm/watchyourlan.yaml` | ✅ Running |
| **watchtower-stack** | `common/watchtower-full.yaml` | ✅ Running |
| **webcheck-stack** | `hosts/vms/homelab-vm/webcheck.yaml` | ✅ Running |

### Raspberry Pi 5 (ep=443395) — 4 Stacks

| Stack Name | Config Path | Status |
|------------|-------------|--------|
| **diun-stack** | `hosts/edge/rpi5-vish/diun.yaml` | ✅ Running |
| **glances-stack** | `hosts/edge/rpi5-vish/glances.yaml` | ✅ Running |
| **portainer-agent-stack** | `hosts/edge/rpi5-vish/portainer_agent.yaml` | ✅ Running |
| **uptime-kuma-stack** | `hosts/edge/rpi5-vish/uptime-kuma.yaml` | ✅ Running |

## 🚀 GitOps Workflow

### 1. Service Definition
Services are defined using Docker Compose YAML files in the repository:

```yaml
# Example: Atlantis/new-service.yaml
version: '3.8'
services:
  new-service:
    image: example/service:latest
    container_name: new-service
    ports:
      - "8080:8080"
    environment:
      - ENV_VAR=value
    volumes:
      - /volume1/docker/new-service:/data
    restart: unless-stopped
```

### 2. Git Commit & Push
```bash
# Add new service configuration
git add Atlantis/new-service.yaml
git commit -m "Add new service deployment

- Configure new-service with proper volumes
- Set up environment variables
- Enable auto-restart policy"

# Push to trigger GitOps deployment
git push origin main
```

### 3. Automatic Deployment
- Portainer monitors the Git repository for changes
- New commits trigger automatic stack updates
- Services are deployed/updated across the infrastructure
- Health checks verify successful deployment

### 4. Monitoring & Verification
```bash
# Check deployment status
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker compose ls"

# Verify service health
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker ps | grep new-service"
```

## 📁 Repository Structure for GitOps

### Host-Specific Configurations

All stacks use canonical `hosts/` paths. The root-level legacy directories (`Atlantis/`, `Calypso/`, etc.) are symlinks kept only for backwards compatibility — do not use them for new stacks.

```
homelab/
├── hosts/
│   ├── synology/
│   │   ├── atlantis/           # Synology DS1823xs+ (Primary NAS)
│   │   │   ├── arr-suite/      # Media automation stack
│   │   │   ├── immich/         # Photo management
│   │   │   ├── ollama/         # AI/LLM services
│   │   │   └── *.yaml          # Individual service configs
│   │   └── calypso/            # Synology DS723+ (Secondary NAS)
│   │       ├── authentik/      # SSO platform
│   │       ├── immich/         # Photo backup
│   │       ├── paperless/      # Document management
│   │       └── *.yaml          # Service configurations
│   ├── physical/
│   │   └── concord-nuc/        # Intel NUC (Edge Computing)
│   │       ├── homeassistant.yaml
│   │       ├── invidious/      # YouTube frontend
│   │       └── *.yaml
│   ├── vms/
│   │   └── homelab-vm/         # Proxmox VM
│   │       ├── monitoring.yaml # Prometheus + Grafana
│   │       └── *.yaml          # Cloud service configs
│   └── edge/
│       └── rpi5-vish/          # Raspberry Pi 5 (IoT/Edge)
│           └── *.yaml
└── common/                     # Shared configurations
    └── watchtower-full.yaml    # Auto-update (all hosts)
```

### Service Categories
- **Media & Entertainment**: Plex, Jellyfin, *arr suite, Immich
- **Development & DevOps**: Gitea, Portainer, monitoring stack
- **Productivity**: PaperlessNGX, Joplin, Syncthing
- **Network & Infrastructure**: AdGuard, Nginx Proxy Manager, Authentik
- **Communication**: Stoatchat, Matrix, Jitsi
- **Utilities**: Watchtower, theme-park, IT Tools

## 🔧 Service Management Operations

### Adding a New Service

1. **Create Service Configuration**
```bash
# Create new service file
cat > Atlantis/new-service.yaml << 'EOF'
version: '3.8'
services:
  new-service:
    image: example/service:latest
    container_name: new-service
    ports:
      - "8080:8080"
    volumes:
      - /volume1/docker/new-service:/data
    restart: unless-stopped
EOF
```

2. **Commit and Deploy**
```bash
git add Atlantis/new-service.yaml
git commit -m "Add new-service deployment"
git push origin main
```

3. **Verify Deployment**
```bash
# Check if stack was created
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker compose ls | grep new-service"

# Verify container is running
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker ps | grep new-service"
```

### Updating an Existing Service

1. **Modify Configuration**
```bash
# Edit existing service
nano Atlantis/existing-service.yaml
```

2. **Commit Changes**
```bash
git add Atlantis/existing-service.yaml
git commit -m "Update existing-service configuration

- Upgrade to latest image version
- Add new environment variables
- Update volume mounts"
git push origin main
```

3. **Monitor Update**
- Portainer will automatically pull changes
- Service will be redeployed with new configuration
- Check Portainer UI for deployment status

### Removing a Service

1. **Remove Configuration File**
```bash
git rm Atlantis/old-service.yaml
git commit -m "Remove old-service deployment"
git push origin main
```

2. **Manual Cleanup (if needed)**
```bash
# Remove any persistent volumes or data
ssh -p 60000 vish@192.168.0.200 "sudo rm -rf /volume1/docker/old-service"
```

## 🔍 Monitoring & Troubleshooting

### GitOps Health Checks

#### Check Portainer Status
```bash
# Verify Portainer is running
curl -k -s "https://192.168.0.200:9443/api/system/status"

# Check container status
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker ps | grep portainer"
```

#### Verify Git Sync Status
```bash
# Check if Portainer can access Git repository
# (Check via Portainer UI: Stacks → Repository sync status)

# Verify latest commits are reflected
git log --oneline -5
```

#### Monitor Stack Deployments
```bash
# List all active stacks
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker compose ls"

# Check specific stack status
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker compose -f /path/to/stack.yaml ps"
```

### Common Issues & Solutions

#### Stack Deployment Fails
1. **Check YAML Syntax**
```bash
# Validate YAML syntax
yamllint Atlantis/service.yaml

# Check Docker Compose syntax
docker-compose -f Atlantis/service.yaml config
```

2. **Review Portainer Logs**
```bash
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker logs portainer"
```

3. **Check Resource Constraints**
```bash
# Verify disk space
ssh -p 60000 vish@192.168.0.200 "df -h"

# Check memory usage
ssh -p 60000 vish@192.168.0.200 "free -h"
```

#### Git Repository Access Issues
1. **Verify Repository URL**
2. **Check Authentication credentials**
3. **Confirm network connectivity**

#### Service Won't Start
1. **Check container logs**
```bash
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker logs service-name"
```

2. **Verify port conflicts**
```bash
ssh -p 60000 vish@192.168.0.200 "sudo netstat -tulpn | grep :PORT"
```

3. **Check volume mounts**
```bash
ssh -p 60000 vish@192.168.0.200 "ls -la /volume1/docker/service-name"
```

## 🔐 Security Considerations

### GitOps Security Best Practices
- **Repository Access**: Secure Git repository with appropriate access controls
- **Secrets Management**: Use Docker secrets or external secret management
- **Network Security**: Services deployed on isolated Docker networks
- **Regular Updates**: Watchtower ensures containers stay updated

### Access Control
- **Portainer Authentication**: Multi-user access with role-based permissions
- **SSH Access**: Key-based authentication for server management
- **Service Authentication**: Individual service authentication where applicable

## 📈 Performance & Scaling

### Resource Monitoring
- **Container Metrics**: Monitor CPU, memory, and disk usage
- **Network Performance**: Track bandwidth and connection metrics
- **Storage Utilization**: Monitor disk space across all hosts

### Scaling Strategies
- **Horizontal Scaling**: Deploy services across multiple hosts
- **Load Balancing**: Use Nginx Proxy Manager for traffic distribution
- **Resource Optimization**: Optimize container resource limits

## 🔄 Backup & Disaster Recovery

### GitOps Backup Strategy
- **Repository Backup**: Git repository is the source of truth
- **Configuration Backup**: All service configurations version controlled
- **Data Backup**: Persistent volumes backed up separately

### Recovery Procedures
1. **Service Recovery**: Redeploy from Git repository
2. **Data Recovery**: Restore from backup volumes
3. **Full Infrastructure Recovery**: Bootstrap new hosts with GitOps

## 📚 Related Documentation

- [GITOPS_DEPLOYMENT_GUIDE.md](../GITOPS_DEPLOYMENT_GUIDE.md) - Original deployment guide
- [MONITORING_ARCHITECTURE.md](../MONITORING_ARCHITECTURE.md) - Monitoring setup
- [docs/admin/portainer-backup.md](portainer-backup.md) - Portainer backup procedures
- [docs/runbooks/add-new-service.md](../runbooks/add-new-service.md) - Service deployment runbook

## 🎯 Next Steps

### Short Term
- [ ] Set up automated GitOps health monitoring
- [ ] Create service deployment templates
- [ ] Implement automated testing for configurations

### Medium Term
- [ ] Expand GitOps to additional hosts
- [ ] Implement blue-green deployments
- [ ] Add configuration validation pipelines

### Long Term
- [ ] Migrate to Kubernetes GitOps (ArgoCD/Flux)
- [ ] Implement infrastructure as code (Terraform)
- [ ] Add automated disaster recovery testing

---

**Document Status**: ✅ Active  
**Deployment Method**: GitOps via Portainer EE  
**Last Verified**: March 8, 2026  
**Next Review**: April 8, 2026