# 🔄 GitOps with Portainer

**🟡 Intermediate Guide**

This guide covers the GitOps deployment model used to manage all Docker stacks in the homelab. Portainer automatically syncs with the Git repository to deploy and update services.

## 🎯 Overview

### How It Works

```
┌─────────────┐    push     ┌─────────────┐    poll (5min)    ┌─────────────┐
│   Git Repo  │ ◄────────── │  Developer  │                   │  Portainer  │
│ git.vish.gg │             │             │                   │             │
└─────────────┘             └─────────────┘                   └──────┬──────┘
       │                                                              │
       │ ─────────────────────────────────────────────────────────────┘
       │                          fetch changes
       ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                         Docker Hosts (5 endpoints)                       │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐  │
│  │ Atlantis │  │ Calypso  │  │ Concord  │  │ Homelab  │  │   RPi5   │  │
│  │  NAS     │  │  NAS     │  │  NUC     │  │   VM     │  │          │  │
│  └──────────┘  └──────────┘  └──────────┘  └──────────┘  └──────────┘  │
└─────────────────────────────────────────────────────────────────────────┘
```

### Key Components

| Component | URL/Location | Purpose |
|-----------|--------------|---------|
| **Git Repository** | `https://git.vish.gg/Vish/homelab.git` | Source of truth for all configs |
| **Portainer** | `http://vishinator.synology.me:10000` | Stack deployment & management |
| **Branch** | `refs/heads/main` | Production deployment branch |

---

## 📁 Repository Structure

Stacks are organized by host. The canonical paths are under `hosts/`:

```
homelab/
├── hosts/
│   ├── synology/
│   │   ├── atlantis/          # Atlantis NAS stacks  ← use this path
│   │   └── calypso/           # Calypso NAS stacks   ← use this path
│   ├── physical/
│   │   └── concord-nuc/       # Intel NUC stacks
│   ├── vms/
│   │   └── homelab-vm/        # Proxmox VM stacks
│   └── edge/
│       └── rpi5-vish/         # Raspberry Pi stacks
├── common/                    # Shared configs (watchtower, etc.)
│
│ # Legacy symlinks — DO NOT use for new stacks (see note below)
├── Atlantis -> hosts/synology/atlantis
├── Calypso -> hosts/synology/calypso
├── concord_nuc -> hosts/physical/concord-nuc
├── homelab_vm -> hosts/vms/homelab-vm
└── raspberry-pi-5-vish -> hosts/edge/rpi5-vish
```

> **Note on symlinks:** The root-level symlinks (`Atlantis/`, `Calypso/`, etc.) exist only for
> backwards compatibility and as Git-level convenience aliases. All Portainer stacks across every
> endpoint have been migrated to canonical `hosts/` paths as of March 2026.
>
> **Always use the canonical `hosts/…` path when creating new Portainer stacks.**

---

## ⚙️ Portainer Stack Settings

### GitOps Updates Configuration

Each stack in Portainer has these settings:

| Setting | Recommended | Description |
|---------|-------------|-------------|
| **GitOps updates** | ✅ ON | Enable automatic sync from Git |
| **Mechanism** | Polling | Check Git periodically (vs webhook) |
| **Fetch interval** | `5m` | How often to check for changes |
| **Re-pull image** | ✅ ON* | Pull fresh `:latest` images on deploy |
| **Force redeployment** | ❌ OFF | Only redeploy when files change |

*Enable "Re-pull image" only for stable services using `:latest` tags.

### When Stacks Update

Portainer only redeploys a stack when:
1. The specific compose file for that stack changes in Git
2. A new commit is pushed that modifies the stack's yaml file

**Important**: Commits that don't touch a stack's compose file won't trigger a redeploy for that stack. This is expected behavior - you don't want every stack restarting on every commit.

---

## 🏷️ Image Tag Strategy

### Recommended Tags by Service Type

| Service Type | Tag Strategy | Re-pull Image |
|--------------|--------------|---------------|
| **Monitoring** (node-exporter, glances) | `:latest` | ✅ ON |
| **Utilities** (watchtower, ntfy) | `:latest` | ✅ ON |
| **Privacy frontends** (redlib, proxitok) | `:latest` | ✅ ON |
| **Databases** (postgres, redis) | `:16`, `:7` (pinned) | ❌ OFF |
| **Critical services** (paperless, immich) | `:latest` or pinned | Case by case |
| **Media servers** (plex, jellyfin) | `:latest` | ✅ ON |

### Stacks with Re-pull Enabled

The following stable stacks have "Re-pull image" enabled for automatic updates:

- `glances-stack` (rpi5)
- `uptime-kuma-stack` (rpi5)
- `watchtower-stack` (all hosts)
- `node-exporter-stack` (Calypso, Concord NUC)
- `diun-stack` (all hosts)
- `dozzle-agent-stack` (all hosts)
- `ntfy-stack` (homelab-vm)
- `redlib-stack` (homelab-vm)
- `proxitok-stack` (homelab-vm)
- `monitoring-stack` (homelab-vm)
- `alerting-stack` (homelab-vm)
- `openhands-stack` (homelab-vm)
- `scrutiny-stack` (homelab-vm)
- `scrutiny-collector-stack` (Calypso, Concord NUC)
- `apt-cacher-ng-stack` (Calypso)
- `paperless-stack` (Calypso)
- `paperless-ai-stack` (Calypso)

---

## 📊 Homelab VM Stacks Reference

All 19 stacks on Homelab VM (192.168.0.210) are deployed via GitOps on canonical `hosts/` paths:

| Stack ID | Name | Compose Path | Description |
|----------|------|--------------|-------------|
| 687 | `monitoring-stack` | `hosts/vms/homelab-vm/monitoring.yaml` | Prometheus, Grafana, Node Exporter, SNMP Exporter |
| 500 | `alerting-stack` | `hosts/vms/homelab-vm/alerting.yaml` | Alertmanager, ntfy-bridge, signal-bridge |
| 501 | `openhands-stack` | `hosts/vms/homelab-vm/openhands.yaml` | AI Software Development Agent |
| 572 | `ntfy-stack` | `hosts/vms/homelab-vm/ntfy.yaml` | Push notification server |
| 566 | `signal-api-stack` | `hosts/vms/homelab-vm/signal_api.yaml` | Signal messaging API |
| 574 | `perplexica-stack` | `hosts/vms/homelab-vm/perplexica.yaml` | AI-powered search |
| 571 | `redlib-stack` | `hosts/vms/homelab-vm/redlib.yaml` | Reddit privacy frontend |
| 570 | `proxitok-stack` | `hosts/vms/homelab-vm/proxitok.yaml` | TikTok privacy frontend |
| 561 | `binternet-stack` | `hosts/vms/homelab-vm/binternet.yaml` | Pinterest privacy frontend |
| 562 | `hoarder-karakeep-stack` | `hosts/vms/homelab-vm/hoarder.yaml` | Bookmark manager |
| 567 | `archivebox-stack` | `hosts/vms/homelab-vm/archivebox.yaml` | Web archive |
| 568 | `drawio-stack` | `hosts/vms/homelab-vm/drawio.yml` | Diagramming tool |
| 563 | `webcheck-stack` | `hosts/vms/homelab-vm/webcheck.yaml` | Website analysis |
| 564 | `watchyourlan-stack` | `hosts/vms/homelab-vm/watchyourlan.yaml` | LAN monitoring |
| 565 | `syncthing-stack` | `hosts/vms/homelab-vm/syncthing.yml` | File synchronization |
| 684 | `diun-stack` | `hosts/vms/homelab-vm/diun.yaml` | Docker image update notifier |
| 685 | `dozzle-agent-stack` | `hosts/vms/homelab-vm/dozzle-agent.yaml` | Container log aggregation agent |
| 686 | `scrutiny-stack` | `hosts/vms/homelab-vm/scrutiny.yaml` | Disk S.M.A.R.T. monitoring |
| 470 | `watchtower-stack` | `common/watchtower-full.yaml` | Auto container updates |

### Monitoring & Alerting Architecture

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                           HOMELAB VM MONITORING                              │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ┌─────────────┐    scrape    ┌─────────────┐    query    ┌─────────────┐  │
│  │ Node Export │──────────────▶│  Prometheus │◀────────────│   Grafana   │  │
│  │ SNMP Export │              │   :9090     │              │    :3300    │  │
│  └─────────────┘              └──────┬──────┘              └─────────────┘  │
│                                      │                                       │
│                                      │ alerts                                │
│                                      ▼                                       │
│                             ┌─────────────────┐                              │
│                             │  Alertmanager   │                              │
│                             │     :9093       │                              │
│                             └────────┬────────┘                              │
│                                      │                                       │
│               ┌──────────────────────┼──────────────────────┐                │
│               │                      │                      │                │
│               ▼                      ▼                      ▼                │
│        ┌─────────────┐        ┌─────────────┐        ┌─────────────┐        │
│        │ ntfy-bridge │        │signal-bridge│        │   (future)  │        │
│        │    :5001    │        │    :5000    │        │             │        │
│        └──────┬──────┘        └──────┬──────┘        └─────────────┘        │
│               │                      │                                       │
│               ▼                      ▼                                       │
│        ┌─────────────┐        ┌─────────────┐                               │
│        │    ntfy     │        │ Signal API  │                               │
│        │   server    │        │    :8080    │                               │
│        └─────────────┘        └─────────────┘                               │
│               │                      │                                       │
│               ▼                      ▼                                       │
│          📱 iOS/Android         📱 Signal App                                │
└─────────────────────────────────────────────────────────────────────────────┘
```

---

## 🔧 Managing Stacks

### Adding a New Stack

1. **Create the compose file** in the appropriate host directory:
   ```bash
   cd hosts/synology/calypso/
   vim new-service.yaml
   ```

2. **Commit and push**:
   ```bash
   git add new-service.yaml
   git commit -m "Add new-service to Calypso"
   git push origin main
   ```

3. **Create stack in Portainer**:
   - Go to Stacks → Add stack
   - Select "Repository"
   - Repository URL: `https://git.vish.gg/Vish/homelab.git`
   - Reference: `refs/heads/main`
   - Compose path: `hosts/synology/calypso/new-service.yaml` (always use canonical `hosts/` path)
   - Enable GitOps updates with 5m polling

### Updating an Existing Stack

1. **Edit the compose file**:
   ```bash
   vim hosts/synology/calypso/existing-service.yaml
   ```

2. **Commit and push**:
   ```bash
   git commit -am "Update existing-service configuration"
   git push origin main
   ```

3. **Wait for auto-sync** (up to 5 minutes) or manually click "Pull and redeploy" in Portainer

### Force Immediate Update

In Portainer UI:
1. Go to the stack
2. Click "Pull and redeploy"
3. Optionally enable "Re-pull image" for this deployment

Via API:
```bash
curl -X PUT \
  -H "X-API-Key: YOUR_API_KEY" \
  "http://vishinator.synology.me:10000/api/stacks/{id}/git/redeploy?endpointId={endpointId}" \
  -d '{"pullImage":true,"repositREDACTED_APP_PASSWORD":"refs/heads/main","prune":false}'
```

### Creating a GitOps Stack via API

To create a new GitOps stack from the repository:

```bash
curl -X POST \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  "http://vishinator.synology.me:10000/api/stacks/create/standalone/repository?endpointId=443399" \
  -d '{
    "name": "my-new-stack",
    "repositoryURL": "https://git.vish.gg/Vish/homelab.git",
    "repositREDACTED_APP_PASSWORD": "refs/heads/main",
    "composeFile": "hosts/vms/homelab-vm/my-service.yaml",
    "repositoREDACTED_APP_PASSWORD": true,
    "reREDACTED_APP_PASSWORD": "",
    "reREDACTED_APP_PASSWORD": "YOUR_GIT_TOKEN",
    "autoUpdate": {
      "interval": "5m",
      "forceUpdate": false,
      "forcePullImage": false
    }
  }'
```

**Endpoint IDs:**
| Endpoint | ID |
|----------|-----|
| Atlantis | 2 |
| Calypso | 443397 |
| Homelab VM | 443399 |
| RPi5 | 443395 |
| Concord NUC | 443398 |

---

## 📊 Monitoring Sync Status

### Check Stack Versions

Each stack shows its current Git commit hash. Compare with the repo:

```bash
# Get current repo HEAD
git log -1 --format="%H"

# Check in Portainer
# Stack → GitConfig → ConfigHash should match
```

### Common Sync States

| ConfigHash matches HEAD | Stack files changed | Result |
|------------------------|---------------------|--------|
| ✅ Yes | N/A | Up to date |
| ❌ No | ✅ Yes | Will update on next poll |
| ❌ No | ❌ No | Expected - stack unchanged |

### Troubleshooting Sync Issues

**Stack not updating:**
1. Check if the specific compose file changed (not just any file)
2. Verify Git credentials in Portainer are valid
3. Check Portainer logs for fetch errors
4. Try manual "Pull and redeploy"

**Wrong version deployed:**
1. Verify the branch is `refs/heads/main`
2. Check compose file path matches (watch for symlinks)
3. Clear Portainer's git cache by recreating the stack

---

## 🔐 Git Authentication

Stacks use a shared Git credential configured in Portainer:

| Setting | Value |
|---------|-------|
| **Credential ID** | 1 |
| **Repository** | `https://git.vish.gg/Vish/homelab.git` |
| **Auth Type** | Token-based |

To update credentials:
1. Portainer → Settings → Credentials
2. Update the Git credential
3. All stacks using that credential will use the new token

---

## 📋 Best Practices

### Do ✅

- Use descriptive commit messages for stack changes
- Test compose files locally before pushing
- Keep one service per compose file when possible
- Use canonical `hosts/…` paths in Portainer for new stacks (not symlink paths)
- Enable re-pull for stable `:latest` services

### Don't ❌

- Force redeployment (causes unnecessary restarts)
- Use `latest` tag for databases
- Push broken compose files to main
- Manually edit stacks in Portainer (changes will be overwritten)

---

## 🔗 Related Documentation

- **[Deployment Guide](deployment.md)** - How to create new services
- **[Monitoring Setup](monitoring.md)** - Track stack health
- **[Troubleshooting](../troubleshooting/common-issues.md)** - Common problems

---

*Last updated: March 2026*
