homelab-optimized/docs/admin/GITOPS_COMPREHENSIVE_GUIDE.md

GitOps Deployment Comprehensive Guide

Last Updated: February 14, 2026

🎯 Overview

This homelab infrastructure is deployed using GitOps methodology with Portainer Enterprise Edition as the orchestration platform. All services are defined as Docker Compose files in this Git repository and automatically deployed across multiple hosts.

🏗️ GitOps Architecture

Core Components

• Git Repository: Source of truth for all infrastructure configurations
• Portainer EE: GitOps orchestration and container management (v2.33.7)
• Docker Compose: Service definition and deployment format
• Multi-Host Deployment: Services distributed across Synology NAS, VMs, and edge devices

Current Deployment Status

Verified Active Stacks: 18 compose stacks deployed via GitOps Total Containers: 50+ containers across infrastructure Management Interface: https://192.168.0.200:9443 (Portainer EE)

📊 Active GitOps Deployments

Atlantis (Primary NAS) - 18 Stacks Active

Stack Name	Containers	Config Path	Status
arr-stack	18	Atlantis/arr-suite/arrs-compose.yaml	✅ Running
baikal-stack	1	Atlantis/baikal/baikal.yaml	✅ Running
dokuwiki-stack	1	Atlantis/dokuwiki.yml	✅ Running
dyndns-updater-stack	3	Atlantis/dynamicdnsupdater.yaml	✅ Running
fenrus-stack	1	Atlantis/fenrus.yaml	✅ Running
homarr-stack	1	hosts/synology/atlantis/homarr.yaml	✅ Running
immich-stack	4	Atlantis/immich/docker-compose.yml	✅ Running
iperf3-stack	1	Atlantis/iperf3.yaml	✅ Running
it_tools-stack	1	Atlantis/it_tools.yml	✅ Running
jitsi	5	Atlantis/jitsi/jitsi.yml	✅ Running
joplin-stack	2	Atlantis/joplin.yml	✅ Running
node-exporter-stack	2	Atlantis/grafana_prometheus/atlantis_node_exporter.yaml	✅ Running
ollama	2	Atlantis/ollama/docker-compose.yml	✅ Running
syncthing-stack	1	Atlantis/syncthing.yml	✅ Running
theme-park	1	Atlantis/theme-park/theme-park.yaml	✅ Running
vaultwarden-stack	2	Atlantis/vaultwarden.yaml	✅ Running
watchtower-stack	1	common/watchtower-full.yaml	✅ Running
youtubedl	1	Atlantis/youtubedl.yaml	✅ Running

Other Hosts

• Calypso: Synology DS723+ (Secondary NAS)
• Homelab VM: Proxmox VM (Cloud Services)
• Concord NUC: Intel NUC (Edge Computing)
• Raspberry Pi 5: ARM64 (IoT/Edge)

🚀 GitOps Workflow

1. Service Definition

Services are defined using Docker Compose YAML files in the repository:

# Example: Atlantis/new-service.yaml
version: '3.8'
services:
  new-service:
    image: example/service:latest
    container_name: new-service
    ports:
      - "8080:8080"
    environment:
      - ENV_VAR=value
    volumes:
      - /volume1/docker/new-service:/data
    restart: unless-stopped

2. Git Commit & Push

# Add new service configuration
git add Atlantis/new-service.yaml
git commit -m "Add new service deployment

- Configure new-service with proper volumes
- Set up environment variables
- Enable auto-restart policy"

# Push to trigger GitOps deployment
git push origin main

3. Automatic Deployment

• Portainer monitors the Git repository for changes
• New commits trigger automatic stack updates
• Services are deployed/updated across the infrastructure
• Health checks verify successful deployment

4. Monitoring & Verification

# Check deployment status
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker compose ls"

# Verify service health
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker ps | grep new-service"

📁 Repository Structure for GitOps

Host-Specific Configurations

homelab/
├── Atlantis/                    # Synology DS1823xs+ (Primary NAS)
│   ├── arr-suite/              # Media automation stack
│   ├── immich/                 # Photo management
│   ├── ollama/                 # AI/LLM services
│   └── *.yaml                  # Individual service configs
├── Calypso/                    # Synology DS723+ (Secondary NAS)
│   ├── grafana_prometheus/     # Monitoring stack
│   ├── immich/                 # Photo backup
│   └── *.yaml                  # Service configurations
├── homelab_vm/                 # Proxmox VM (Cloud Services)
│   ├── monitoring.yaml         # Production monitoring
│   ├── stoatchat.yaml         # Chat platform
│   └── *.yaml                  # Cloud service configs
├── concord_nuc/                # Intel NUC (Edge Computing)
│   ├── homeassistant.yaml     # Home automation
│   ├── plex.yaml              # Media server
│   └── *.yaml                  # Edge service configs
└── common/                     # Shared configurations
    └── watchtower-*.yaml       # Auto-update services

Service Categories

• Media & Entertainment: Plex, Jellyfin, *arr suite, Immich
• Development & DevOps: Gitea, Portainer, monitoring stack
• Productivity: PaperlessNGX, Joplin, Syncthing
• Network & Infrastructure: AdGuard, Nginx Proxy Manager, Authentik
• Communication: Stoatchat, Matrix, Jitsi
• Utilities: Watchtower, theme-park, IT Tools

🔧 Service Management Operations

Adding a New Service

1. Create Service Configuration

# Create new service file
cat > Atlantis/new-service.yaml << 'EOF'
version: '3.8'
services:
  new-service:
    image: example/service:latest
    container_name: new-service
    ports:
      - "8080:8080"
    volumes:
      - /volume1/docker/new-service:/data
    restart: unless-stopped
EOF

2. Commit and Deploy

git add Atlantis/new-service.yaml
git commit -m "Add new-service deployment"
git push origin main

3. Verify Deployment

# Check if stack was created
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker compose ls | grep new-service"

# Verify container is running
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker ps | grep new-service"

Updating an Existing Service

1. Modify Configuration

# Edit existing service
nano Atlantis/existing-service.yaml

2. Commit Changes

git add Atlantis/existing-service.yaml
git commit -m "Update existing-service configuration

- Upgrade to latest image version
- Add new environment variables
- Update volume mounts"
git push origin main

3. Monitor Update

• Portainer will automatically pull changes
• Service will be redeployed with new configuration
• Check Portainer UI for deployment status

Removing a Service

1. Remove Configuration File

git rm Atlantis/old-service.yaml
git commit -m "Remove old-service deployment"
git push origin main

2. Manual Cleanup (if needed)

# Remove any persistent volumes or data
ssh -p 60000 vish@192.168.0.200 "sudo rm -rf /volume1/docker/old-service"

🔍 Monitoring & Troubleshooting

GitOps Health Checks

Check Portainer Status

# Verify Portainer is running
curl -k -s "https://192.168.0.200:9443/api/system/status"

# Check container status
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker ps | grep portainer"

Verify Git Sync Status

# Check if Portainer can access Git repository
# (Check via Portainer UI: Stacks → Repository sync status)

# Verify latest commits are reflected
git log --oneline -5

Monitor Stack Deployments

# List all active stacks
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker compose ls"

# Check specific stack status
ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker compose -f /path/to/stack.yaml ps"

Common Issues & Solutions

Stack Deployment Fails

1. Check YAML Syntax

# Validate YAML syntax
yamllint Atlantis/service.yaml

# Check Docker Compose syntax
docker-compose -f Atlantis/service.yaml config

2. Review Portainer Logs

ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker logs portainer"

3. Check Resource Constraints

# Verify disk space
ssh -p 60000 vish@192.168.0.200 "df -h"

# Check memory usage
ssh -p 60000 vish@192.168.0.200 "free -h"

Git Repository Access Issues

1. Verify Repository URL
2. Check Authentication credentials
3. Confirm network connectivity

Service Won't Start

1. Check container logs

ssh -p 60000 vish@192.168.0.200 "sudo /usr/local/bin/docker logs service-name"

2. Verify port conflicts

ssh -p 60000 vish@192.168.0.200 "sudo netstat -tulpn | grep :PORT"

3. Check volume mounts

ssh -p 60000 vish@192.168.0.200 "ls -la /volume1/docker/service-name"

🔐 Security Considerations

GitOps Security Best Practices

• Repository Access: Secure Git repository with appropriate access controls
• Secrets Management: Use Docker secrets or external secret management
• Network Security: Services deployed on isolated Docker networks
• Regular Updates: Watchtower ensures containers stay updated

Access Control

• Portainer Authentication: Multi-user access with role-based permissions
• SSH Access: Key-based authentication for server management
• Service Authentication: Individual service authentication where applicable

📈 Performance & Scaling

Resource Monitoring

• Container Metrics: Monitor CPU, memory, and disk usage
• Network Performance: Track bandwidth and connection metrics
• Storage Utilization: Monitor disk space across all hosts

Scaling Strategies

• Horizontal Scaling: Deploy services across multiple hosts
• Load Balancing: Use Nginx Proxy Manager for traffic distribution
• Resource Optimization: Optimize container resource limits

🔄 Backup & Disaster Recovery

GitOps Backup Strategy

• Repository Backup: Git repository is the source of truth
• Configuration Backup: All service configurations version controlled
• Data Backup: Persistent volumes backed up separately

Recovery Procedures

1. Service Recovery: Redeploy from Git repository
2. Data Recovery: Restore from backup volumes
3. Full Infrastructure Recovery: Bootstrap new hosts with GitOps

📚 Related Documentation

• GITOPS_DEPLOYMENT_GUIDE.md - Original deployment guide
• MONITORING_ARCHITECTURE.md - Monitoring setup
• docs/admin/portainer-backup.md - Portainer backup procedures
• docs/runbooks/add-new-service.md - Service deployment runbook

🎯 Next Steps

Short Term

• Set up automated GitOps health monitoring
• Create service deployment templates
• Implement automated testing for configurations

Medium Term

• Expand GitOps to additional hosts
• Implement blue-green deployments
• Add configuration validation pipelines

Long Term

• Migrate to Kubernetes GitOps (ArgoCD/Flux)
• Implement infrastructure as code (Terraform)
• Add automated disaster recovery testing

---

Document Status: ✅ Active
Deployment Method: GitOps via Portainer EE
Last Verified: February 14, 2026
Next Review: March 14, 2026