375 lines
16 KiB
Markdown
375 lines
16 KiB
Markdown
# 🔄 GitOps with Portainer
|
|
|
|
**🟡 Intermediate Guide**
|
|
|
|
This guide covers the GitOps deployment model used to manage all Docker stacks in the homelab. Portainer automatically syncs with the Git repository to deploy and update services.
|
|
|
|
## 🎯 Overview
|
|
|
|
### How It Works
|
|
|
|
```
|
|
┌─────────────┐ push ┌─────────────┐ poll (5min) ┌─────────────┐
|
|
│ Git Repo │ ◄────────── │ Developer │ │ Portainer │
|
|
│ git.vish.gg │ │ │ │ │
|
|
└─────────────┘ └─────────────┘ └──────┬──────┘
|
|
│ │
|
|
│ ─────────────────────────────────────────────────────────────┘
|
|
│ fetch changes
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ Docker Hosts (5 endpoints) │
|
|
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
|
│ │ Atlantis │ │ Calypso │ │ Concord │ │ Homelab │ │ RPi5 │ │
|
|
│ │ NAS │ │ NAS │ │ NUC │ │ VM │ │ │ │
|
|
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Key Components
|
|
|
|
| Component | URL/Location | Purpose |
|
|
|-----------|--------------|---------|
|
|
| **Git Repository** | `https://git.vish.gg/Vish/homelab.git` | Source of truth for all configs |
|
|
| **Portainer** | `http://vishinator.synology.me:10000` | Stack deployment & management |
|
|
| **Branch** | `refs/heads/main` | Production deployment branch |
|
|
|
|
---
|
|
|
|
## 📁 Repository Structure
|
|
|
|
Stacks are organized by host. The canonical paths are under `hosts/`:
|
|
|
|
```
|
|
homelab/
|
|
├── hosts/
|
|
│ ├── synology/
|
|
│ │ ├── atlantis/ # Atlantis NAS stacks ← use this path
|
|
│ │ └── calypso/ # Calypso NAS stacks ← use this path
|
|
│ ├── physical/
|
|
│ │ └── concord-nuc/ # Intel NUC stacks
|
|
│ ├── vms/
|
|
│ │ └── homelab-vm/ # Proxmox VM stacks
|
|
│ └── edge/
|
|
│ └── rpi5-vish/ # Raspberry Pi stacks
|
|
├── common/ # Shared configs (watchtower, etc.)
|
|
│
|
|
│ # Legacy symlinks — DO NOT use for new stacks (see note below)
|
|
├── Atlantis -> hosts/synology/atlantis
|
|
├── Calypso -> hosts/synology/calypso
|
|
├── concord_nuc -> hosts/physical/concord-nuc
|
|
├── homelab_vm -> hosts/vms/homelab-vm
|
|
└── raspberry-pi-5-vish -> hosts/edge/rpi5-vish
|
|
```
|
|
|
|
> **Note on symlinks:** The root-level symlinks (`Atlantis/`, `Calypso/`, etc.) exist only for
|
|
> backwards compatibility and as Git-level convenience aliases. All Portainer stacks across every
|
|
> endpoint have been migrated to canonical `hosts/` paths as of March 2026.
|
|
>
|
|
> **Always use the canonical `hosts/…` path when creating new Portainer stacks.**
|
|
|
|
---
|
|
|
|
## ⚙️ Portainer Stack Settings
|
|
|
|
### GitOps Updates Configuration
|
|
|
|
Each stack in Portainer has these settings:
|
|
|
|
| Setting | Recommended | Description |
|
|
|---------|-------------|-------------|
|
|
| **GitOps updates** | ✅ ON | Enable automatic sync from Git |
|
|
| **Mechanism** | Polling | Check Git periodically (vs webhook) |
|
|
| **Fetch interval** | `5m` | How often to check for changes |
|
|
| **Re-pull image** | ✅ ON* | Pull fresh `:latest` images on deploy |
|
|
| **Force redeployment** | ❌ OFF | Only redeploy when files change |
|
|
|
|
*Enable "Re-pull image" only for stable services using `:latest` tags.
|
|
|
|
### When Stacks Update
|
|
|
|
Portainer only redeploys a stack when:
|
|
1. The specific compose file for that stack changes in Git
|
|
2. A new commit is pushed that modifies the stack's yaml file
|
|
|
|
**Important**: Commits that don't touch a stack's compose file won't trigger a redeploy for that stack. This is expected behavior - you don't want every stack restarting on every commit.
|
|
|
|
---
|
|
|
|
## 🏷️ Image Tag Strategy
|
|
|
|
### Recommended Tags by Service Type
|
|
|
|
| Service Type | Tag Strategy | Re-pull Image |
|
|
|--------------|--------------|---------------|
|
|
| **Monitoring** (node-exporter, glances) | `:latest` | ✅ ON |
|
|
| **Utilities** (watchtower, ntfy) | `:latest` | ✅ ON |
|
|
| **Privacy frontends** (redlib, proxitok) | `:latest` | ✅ ON |
|
|
| **Databases** (postgres, redis) | `:16`, `:7` (pinned) | ❌ OFF |
|
|
| **Critical services** (paperless, immich) | `:latest` or pinned | Case by case |
|
|
| **Media servers** (plex, jellyfin) | `:latest` | ✅ ON |
|
|
|
|
### Stacks with Re-pull Enabled
|
|
|
|
The following stable stacks have "Re-pull image" enabled for automatic updates:
|
|
|
|
- `glances-stack` (rpi5)
|
|
- `uptime-kuma-stack` (rpi5)
|
|
- `watchtower-stack` (all hosts)
|
|
- `node-exporter-stack` (Calypso, Concord NUC)
|
|
- `diun-stack` (all hosts)
|
|
- `dozzle-agent-stack` (all hosts)
|
|
- `ntfy-stack` (homelab-vm)
|
|
- `redlib-stack` (homelab-vm)
|
|
- `proxitok-stack` (homelab-vm)
|
|
- `monitoring-stack` (homelab-vm)
|
|
- `alerting-stack` (homelab-vm)
|
|
- `openhands-stack` (homelab-vm)
|
|
- `scrutiny-stack` (homelab-vm)
|
|
- `scrutiny-collector-stack` (Calypso, Concord NUC)
|
|
- `apt-cacher-ng-stack` (Calypso)
|
|
- `paperless-stack` (Calypso)
|
|
- `paperless-ai-stack` (Calypso)
|
|
|
|
---
|
|
|
|
## 📊 Homelab VM Stacks Reference
|
|
|
|
All 19 stacks on Homelab VM (192.168.0.210) are deployed via GitOps on canonical `hosts/` paths:
|
|
|
|
| Stack ID | Name | Compose Path | Description |
|
|
|----------|------|--------------|-------------|
|
|
| 687 | `monitoring-stack` | `hosts/vms/homelab-vm/monitoring.yaml` | Prometheus, Grafana, Node Exporter, SNMP Exporter |
|
|
| 500 | `alerting-stack` | `hosts/vms/homelab-vm/alerting.yaml` | Alertmanager, ntfy-bridge, signal-bridge |
|
|
| 501 | `openhands-stack` | `hosts/vms/homelab-vm/openhands.yaml` | AI Software Development Agent |
|
|
| 572 | `ntfy-stack` | `hosts/vms/homelab-vm/ntfy.yaml` | Push notification server |
|
|
| 566 | `signal-api-stack` | `hosts/vms/homelab-vm/signal_api.yaml` | Signal messaging API |
|
|
| 574 | `perplexica-stack` | `hosts/vms/homelab-vm/perplexica.yaml` | AI-powered search |
|
|
| 571 | `redlib-stack` | `hosts/vms/homelab-vm/redlib.yaml` | Reddit privacy frontend |
|
|
| 570 | `proxitok-stack` | `hosts/vms/homelab-vm/proxitok.yaml` | TikTok privacy frontend |
|
|
| 561 | `binternet-stack` | `hosts/vms/homelab-vm/binternet.yaml` | Pinterest privacy frontend |
|
|
| 562 | `hoarder-karakeep-stack` | `hosts/vms/homelab-vm/hoarder.yaml` | Bookmark manager |
|
|
| 567 | `archivebox-stack` | `hosts/vms/homelab-vm/archivebox.yaml` | Web archive |
|
|
| 568 | `drawio-stack` | `hosts/vms/homelab-vm/drawio.yml` | Diagramming tool |
|
|
| 563 | `webcheck-stack` | `hosts/vms/homelab-vm/webcheck.yaml` | Website analysis |
|
|
| 564 | `watchyourlan-stack` | `hosts/vms/homelab-vm/watchyourlan.yaml` | LAN monitoring |
|
|
| 565 | `syncthing-stack` | `hosts/vms/homelab-vm/syncthing.yml` | File synchronization |
|
|
| 684 | `diun-stack` | `hosts/vms/homelab-vm/diun.yaml` | Docker image update notifier |
|
|
| 685 | `dozzle-agent-stack` | `hosts/vms/homelab-vm/dozzle-agent.yaml` | Container log aggregation agent |
|
|
| 686 | `scrutiny-stack` | `hosts/vms/homelab-vm/scrutiny.yaml` | Disk S.M.A.R.T. monitoring |
|
|
| 470 | `watchtower-stack` | `common/watchtower-full.yaml` | Auto container updates |
|
|
|
|
### Monitoring & Alerting Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ HOMELAB VM MONITORING │
|
|
├─────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ ┌─────────────┐ scrape ┌─────────────┐ query ┌─────────────┐ │
|
|
│ │ Node Export │──────────────▶│ Prometheus │◀────────────│ Grafana │ │
|
|
│ │ SNMP Export │ │ :9090 │ │ :3300 │ │
|
|
│ └─────────────┘ └──────┬──────┘ └─────────────┘ │
|
|
│ │ │
|
|
│ │ alerts │
|
|
│ ▼ │
|
|
│ ┌─────────────────┐ │
|
|
│ │ Alertmanager │ │
|
|
│ │ :9093 │ │
|
|
│ └────────┬────────┘ │
|
|
│ │ │
|
|
│ ┌──────────────────────┼──────────────────────┐ │
|
|
│ │ │ │ │
|
|
│ ▼ ▼ ▼ │
|
|
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
|
│ │ ntfy-bridge │ │signal-bridge│ │ (future) │ │
|
|
│ │ :5001 │ │ :5000 │ │ │ │
|
|
│ └──────┬──────┘ └──────┬──────┘ └─────────────┘ │
|
|
│ │ │ │
|
|
│ ▼ ▼ │
|
|
│ ┌─────────────┐ ┌─────────────┐ │
|
|
│ │ ntfy │ │ Signal API │ │
|
|
│ │ server │ │ :8080 │ │
|
|
│ └─────────────┘ └─────────────┘ │
|
|
│ │ │ │
|
|
│ ▼ ▼ │
|
|
│ 📱 iOS/Android 📱 Signal App │
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## 🔧 Managing Stacks
|
|
|
|
### Adding a New Stack
|
|
|
|
1. **Create the compose file** in the appropriate host directory:
|
|
```bash
|
|
cd hosts/synology/calypso/
|
|
vim new-service.yaml
|
|
```
|
|
|
|
2. **Commit and push**:
|
|
```bash
|
|
git add new-service.yaml
|
|
git commit -m "Add new-service to Calypso"
|
|
git push origin main
|
|
```
|
|
|
|
3. **Create stack in Portainer**:
|
|
- Go to Stacks → Add stack
|
|
- Select "Repository"
|
|
- Repository URL: `https://git.vish.gg/Vish/homelab.git`
|
|
- Reference: `refs/heads/main`
|
|
- Compose path: `hosts/synology/calypso/new-service.yaml` (always use canonical `hosts/` path)
|
|
- Enable GitOps updates with 5m polling
|
|
|
|
### Updating an Existing Stack
|
|
|
|
1. **Edit the compose file**:
|
|
```bash
|
|
vim hosts/synology/calypso/existing-service.yaml
|
|
```
|
|
|
|
2. **Commit and push**:
|
|
```bash
|
|
git commit -am "Update existing-service configuration"
|
|
git push origin main
|
|
```
|
|
|
|
3. **Wait for auto-sync** (up to 5 minutes) or manually click "Pull and redeploy" in Portainer
|
|
|
|
### Force Immediate Update
|
|
|
|
In Portainer UI:
|
|
1. Go to the stack
|
|
2. Click "Pull and redeploy"
|
|
3. Optionally enable "Re-pull image" for this deployment
|
|
|
|
Via API:
|
|
```bash
|
|
curl -X PUT \
|
|
-H "X-API-Key: YOUR_API_KEY" \
|
|
"http://vishinator.synology.me:10000/api/stacks/{id}/git/redeploy?endpointId={endpointId}" \
|
|
-d '{"pullImage":true,"repositREDACTED_APP_PASSWORD":"refs/heads/main","prune":false}'
|
|
```
|
|
|
|
### Creating a GitOps Stack via API
|
|
|
|
To create a new GitOps stack from the repository:
|
|
|
|
```bash
|
|
curl -X POST \
|
|
-H "X-API-Key: YOUR_API_KEY" \
|
|
-H "Content-Type: application/json" \
|
|
"http://vishinator.synology.me:10000/api/stacks/create/standalone/repository?endpointId=443399" \
|
|
-d '{
|
|
"name": "my-new-stack",
|
|
"repositoryURL": "https://git.vish.gg/Vish/homelab.git",
|
|
"repositREDACTED_APP_PASSWORD": "refs/heads/main",
|
|
"composeFile": "hosts/vms/homelab-vm/my-service.yaml",
|
|
"repositoREDACTED_APP_PASSWORD": true,
|
|
"reREDACTED_APP_PASSWORD": "",
|
|
"reREDACTED_APP_PASSWORD": "YOUR_GIT_TOKEN",
|
|
"autoUpdate": {
|
|
"interval": "5m",
|
|
"forceUpdate": false,
|
|
"forcePullImage": false
|
|
}
|
|
}'
|
|
```
|
|
|
|
**Endpoint IDs:**
|
|
| Endpoint | ID |
|
|
|----------|-----|
|
|
| Atlantis | 2 |
|
|
| Calypso | 443397 |
|
|
| Homelab VM | 443399 |
|
|
| RPi5 | 443395 |
|
|
| Concord NUC | 443398 |
|
|
|
|
---
|
|
|
|
## 📊 Monitoring Sync Status
|
|
|
|
### Check Stack Versions
|
|
|
|
Each stack shows its current Git commit hash. Compare with the repo:
|
|
|
|
```bash
|
|
# Get current repo HEAD
|
|
git log -1 --format="%H"
|
|
|
|
# Check in Portainer
|
|
# Stack → GitConfig → ConfigHash should match
|
|
```
|
|
|
|
### Common Sync States
|
|
|
|
| ConfigHash matches HEAD | Stack files changed | Result |
|
|
|------------------------|---------------------|--------|
|
|
| ✅ Yes | N/A | Up to date |
|
|
| ❌ No | ✅ Yes | Will update on next poll |
|
|
| ❌ No | ❌ No | Expected - stack unchanged |
|
|
|
|
### Troubleshooting Sync Issues
|
|
|
|
**Stack not updating:**
|
|
1. Check if the specific compose file changed (not just any file)
|
|
2. Verify Git credentials in Portainer are valid
|
|
3. Check Portainer logs for fetch errors
|
|
4. Try manual "Pull and redeploy"
|
|
|
|
**Wrong version deployed:**
|
|
1. Verify the branch is `refs/heads/main`
|
|
2. Check compose file path matches (watch for symlinks)
|
|
3. Clear Portainer's git cache by recreating the stack
|
|
|
|
---
|
|
|
|
## 🔐 Git Authentication
|
|
|
|
Stacks use a shared Git credential configured in Portainer:
|
|
|
|
| Setting | Value |
|
|
|---------|-------|
|
|
| **Credential ID** | 1 |
|
|
| **Repository** | `https://git.vish.gg/Vish/homelab.git` |
|
|
| **Auth Type** | Token-based |
|
|
|
|
To update credentials:
|
|
1. Portainer → Settings → Credentials
|
|
2. Update the Git credential
|
|
3. All stacks using that credential will use the new token
|
|
|
|
---
|
|
|
|
## 📋 Best Practices
|
|
|
|
### Do ✅
|
|
|
|
- Use descriptive commit messages for stack changes
|
|
- Test compose files locally before pushing
|
|
- Keep one service per compose file when possible
|
|
- Use canonical `hosts/…` paths in Portainer for new stacks (not symlink paths)
|
|
- Enable re-pull for stable `:latest` services
|
|
|
|
### Don't ❌
|
|
|
|
- Force redeployment (causes unnecessary restarts)
|
|
- Use `latest` tag for databases
|
|
- Push broken compose files to main
|
|
- Manually edit stacks in Portainer (changes will be overwritten)
|
|
|
|
---
|
|
|
|
## 🔗 Related Documentation
|
|
|
|
- **[Deployment Guide](deployment.md)** - How to create new services
|
|
- **[Monitoring Setup](monitoring.md)** - Track stack health
|
|
- **[Troubleshooting](../troubleshooting/common-issues.md)** - Common problems
|
|
|
|
---
|
|
|
|
*Last updated: March 2026*
|