Sanitized mirror from private repository - 2026-04-06 03:00:14 UTC

2026-04-06 03:00:14 +00:00
commit 5ad46c4fdb
1414 changed files with 359082 additions and 0 deletions
--- a/docs/admin/gitops.md
+++ b/docs/admin/gitops.md
@@ -0,0 +1,374 @@
+# 🔄 GitOps with Portainer
+
+**🟡 Intermediate Guide**
+
+This guide covers the GitOps deployment model used to manage all Docker stacks in the homelab. Portainer automatically syncs with the Git repository to deploy and update services.
+
+## 🎯 Overview
+
+### How It Works
+
+```
+┌─────────────┐    push     ┌─────────────┐    poll (5min)    ┌─────────────┐
+│   Git Repo  │ ◄────────── │  Developer  │                   │  Portainer  │
+│ git.vish.gg │             │             │                   │             │
+└─────────────┘             └─────────────┘                   └──────┬──────┘
+       │                                                              │
+       │ ─────────────────────────────────────────────────────────────┘
+       │                          fetch changes
+       ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│                         Docker Hosts (5 endpoints)                       │
+│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐  │
+│  │ Atlantis │  │ Calypso  │  │ Concord  │  │ Homelab  │  │   RPi5   │  │
+│  │  NAS     │  │  NAS     │  │  NUC     │  │   VM     │  │          │  │
+│  └──────────┘  └──────────┘  └──────────┘  └──────────┘  └──────────┘  │
+└─────────────────────────────────────────────────────────────────────────┘
+```
+
+### Key Components
+
+| Component | URL/Location | Purpose |
+|-----------|--------------|---------|
+| **Git Repository** | `https://git.vish.gg/Vish/homelab.git` | Source of truth for all configs |
+| **Portainer** | `http://vishinator.synology.me:10000` | Stack deployment & management |
+| **Branch** | `refs/heads/main` | Production deployment branch |
+
+---
+
+## 📁 Repository Structure
+
+Stacks are organized by host. The canonical paths are under `hosts/`:
+
+```
+homelab/
+├── hosts/
+│   ├── synology/
+│   │   ├── atlantis/          # Atlantis NAS stacks  ← use this path
+│   │   └── calypso/           # Calypso NAS stacks   ← use this path
+│   ├── physical/
+│   │   └── concord-nuc/       # Intel NUC stacks
+│   ├── vms/
+│   │   └── homelab-vm/        # Proxmox VM stacks
+│   └── edge/
+│       └── rpi5-vish/         # Raspberry Pi stacks
+├── common/                    # Shared configs (watchtower, etc.)
+│
+│ # Legacy symlinks — DO NOT use for new stacks (see note below)
+├── Atlantis -> hosts/synology/atlantis
+├── Calypso -> hosts/synology/calypso
+├── concord_nuc -> hosts/physical/concord-nuc
+├── homelab_vm -> hosts/vms/homelab-vm
+└── raspberry-pi-5-vish -> hosts/edge/rpi5-vish
+```
+
+> **Note on symlinks:** The root-level symlinks (`Atlantis/`, `Calypso/`, etc.) exist only for
+> backwards compatibility and as Git-level convenience aliases. All Portainer stacks across every
+> endpoint have been migrated to canonical `hosts/` paths as of March 2026.
+>
+> **Always use the canonical `hosts/…` path when creating new Portainer stacks.**
+
+---
+
+## ⚙️ Portainer Stack Settings
+
+### GitOps Updates Configuration
+
+Each stack in Portainer has these settings:
+
+| Setting | Recommended | Description |
+|---------|-------------|-------------|
+| **GitOps updates** | ✅ ON | Enable automatic sync from Git |
+| **Mechanism** | Polling | Check Git periodically (vs webhook) |
+| **Fetch interval** | `5m` | How often to check for changes |
+| **Re-pull image** | ✅ ON* | Pull fresh `:latest` images on deploy |
+| **Force redeployment** | ❌ OFF | Only redeploy when files change |
+
+*Enable "Re-pull image" only for stable services using `:latest` tags.
+
+### When Stacks Update
+
+Portainer only redeploys a stack when:
+1. The specific compose file for that stack changes in Git
+2. A new commit is pushed that modifies the stack's yaml file
+
+**Important**: Commits that don't touch a stack's compose file won't trigger a redeploy for that stack. This is expected behavior - you don't want every stack restarting on every commit.
+
+---
+
+## 🏷️ Image Tag Strategy
+
+### Recommended Tags by Service Type
+
+| Service Type | Tag Strategy | Re-pull Image |
+|--------------|--------------|---------------|
+| **Monitoring** (node-exporter, glances) | `:latest` | ✅ ON |
+| **Utilities** (watchtower, ntfy) | `:latest` | ✅ ON |
+| **Privacy frontends** (redlib, proxitok) | `:latest` | ✅ ON |
+| **Databases** (postgres, redis) | `:16`, `:7` (pinned) | ❌ OFF |
+| **Critical services** (paperless, immich) | `:latest` or pinned | Case by case |
+| **Media servers** (plex, jellyfin) | `:latest` | ✅ ON |
+
+### Stacks with Re-pull Enabled
+
+The following stable stacks have "Re-pull image" enabled for automatic updates:
+
+- `glances-stack` (rpi5)
+- `uptime-kuma-stack` (rpi5)
+- `watchtower-stack` (all hosts)
+- `node-exporter-stack` (Calypso, Concord NUC)
+- `diun-stack` (all hosts)
+- `dozzle-agent-stack` (all hosts)
+- `ntfy-stack` (homelab-vm)
+- `redlib-stack` (homelab-vm)
+- `proxitok-stack` (homelab-vm)
+- `monitoring-stack` (homelab-vm)
+- `alerting-stack` (homelab-vm)
+- `openhands-stack` (homelab-vm)
+- `scrutiny-stack` (homelab-vm)
+- `scrutiny-collector-stack` (Calypso, Concord NUC)
+- `apt-cacher-ng-stack` (Calypso)
+- `paperless-stack` (Calypso)
+- `paperless-ai-stack` (Calypso)
+
+---
+
+## 📊 Homelab VM Stacks Reference
+
+All 19 stacks on Homelab VM (192.168.0.210) are deployed via GitOps on canonical `hosts/` paths:
+
+| Stack ID | Name | Compose Path | Description |
+|----------|------|--------------|-------------|
+| 687 | `monitoring-stack` | `hosts/vms/homelab-vm/monitoring.yaml` | Prometheus, Grafana, Node Exporter, SNMP Exporter |
+| 500 | `alerting-stack` | `hosts/vms/homelab-vm/alerting.yaml` | Alertmanager, ntfy-bridge, signal-bridge |
+| 501 | `openhands-stack` | `hosts/vms/homelab-vm/openhands.yaml` | AI Software Development Agent |
+| 572 | `ntfy-stack` | `hosts/vms/homelab-vm/ntfy.yaml` | Push notification server |
+| 566 | `signal-api-stack` | `hosts/vms/homelab-vm/signal_api.yaml` | Signal messaging API |
+| 574 | `perplexica-stack` | `hosts/vms/homelab-vm/perplexica.yaml` | AI-powered search |
+| 571 | `redlib-stack` | `hosts/vms/homelab-vm/redlib.yaml` | Reddit privacy frontend |
+| 570 | `proxitok-stack` | `hosts/vms/homelab-vm/proxitok.yaml` | TikTok privacy frontend |
+| 561 | `binternet-stack` | `hosts/vms/homelab-vm/binternet.yaml` | Pinterest privacy frontend |
+| 562 | `hoarder-karakeep-stack` | `hosts/vms/homelab-vm/hoarder.yaml` | Bookmark manager |
+| 567 | `archivebox-stack` | `hosts/vms/homelab-vm/archivebox.yaml` | Web archive |
+| 568 | `drawio-stack` | `hosts/vms/homelab-vm/drawio.yml` | Diagramming tool |
+| 563 | `webcheck-stack` | `hosts/vms/homelab-vm/webcheck.yaml` | Website analysis |
+| 564 | `watchyourlan-stack` | `hosts/vms/homelab-vm/watchyourlan.yaml` | LAN monitoring |
+| 565 | `syncthing-stack` | `hosts/vms/homelab-vm/syncthing.yml` | File synchronization |
+| 684 | `diun-stack` | `hosts/vms/homelab-vm/diun.yaml` | Docker image update notifier |
+| 685 | `dozzle-agent-stack` | `hosts/vms/homelab-vm/dozzle-agent.yaml` | Container log aggregation agent |
+| 686 | `scrutiny-stack` | `hosts/vms/homelab-vm/scrutiny.yaml` | Disk S.M.A.R.T. monitoring |
+| 470 | `watchtower-stack` | `common/watchtower-full.yaml` | Auto container updates |
+
+### Monitoring & Alerting Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                           HOMELAB VM MONITORING                              │
+├─────────────────────────────────────────────────────────────────────────────┤
+│                                                                              │
+│  ┌─────────────┐    scrape    ┌─────────────┐    query    ┌─────────────┐  │
+│  │ Node Export │──────────────▶│  Prometheus │◀────────────│   Grafana   │  │
+│  │ SNMP Export │              │   :9090     │              │    :3300    │  │
+│  └─────────────┘              └──────┬──────┘              └─────────────┘  │
+│                                      │                                       │
+│                                      │ alerts                                │
+│                                      ▼                                       │
+│                             ┌─────────────────┐                              │
+│                             │  Alertmanager   │                              │
+│                             │     :9093       │                              │
+│                             └────────┬────────┘                              │
+│                                      │                                       │
+│               ┌──────────────────────┼──────────────────────┐                │
+│               │                      │                      │                │
+│               ▼                      ▼                      ▼                │
+│        ┌─────────────┐        ┌─────────────┐        ┌─────────────┐        │
+│        │ ntfy-bridge │        │signal-bridge│        │   (future)  │        │
+│        │    :5001    │        │    :5000    │        │             │        │
+│        └──────┬──────┘        └──────┬──────┘        └─────────────┘        │
+│               │                      │                                       │
+│               ▼                      ▼                                       │
+│        ┌─────────────┐        ┌─────────────┐                               │
+│        │    ntfy     │        │ Signal API  │                               │
+│        │   server    │        │    :8080    │                               │
+│        └─────────────┘        └─────────────┘                               │
+│               │                      │                                       │
+│               ▼                      ▼                                       │
+│          📱 iOS/Android         📱 Signal App                                │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 🔧 Managing Stacks
+
+### Adding a New Stack
+
+1. **Create the compose file** in the appropriate host directory:
+   ```bash
+   cd hosts/synology/calypso/
+   vim new-service.yaml
+   ```
+
+2. **Commit and push**:
+   ```bash
+   git add new-service.yaml
+   git commit -m "Add new-service to Calypso"
+   git push origin main
+   ```
+
+3. **Create stack in Portainer**:
+   - Go to Stacks → Add stack
+   - Select "Repository"
+   - Repository URL: `https://git.vish.gg/Vish/homelab.git`
+   - Reference: `refs/heads/main`
+   - Compose path: `hosts/synology/calypso/new-service.yaml` (always use canonical `hosts/` path)
+   - Enable GitOps updates with 5m polling
+
+### Updating an Existing Stack
+
+1. **Edit the compose file**:
+   ```bash
+   vim hosts/synology/calypso/existing-service.yaml
+   ```
+
+2. **Commit and push**:
+   ```bash
+   git commit -am "Update existing-service configuration"
+   git push origin main
+   ```
+
+3. **Wait for auto-sync** (up to 5 minutes) or manually click "Pull and redeploy" in Portainer
+
+### Force Immediate Update
+
+In Portainer UI:
+1. Go to the stack
+2. Click "Pull and redeploy"
+3. Optionally enable "Re-pull image" for this deployment
+
+Via API:
+```bash
+curl -X PUT \
+  -H "X-API-Key: YOUR_API_KEY" \
+  "http://vishinator.synology.me:10000/api/stacks/{id}/git/redeploy?endpointId={endpointId}" \
+  -d '{"pullImage":true,"repositREDACTED_APP_PASSWORD":"refs/heads/main","prune":false}'
+```
+
+### Creating a GitOps Stack via API
+
+To create a new GitOps stack from the repository:
+
+```bash
+curl -X POST \
+  -H "X-API-Key: YOUR_API_KEY" \
+  -H "Content-Type: application/json" \
+  "http://vishinator.synology.me:10000/api/stacks/create/standalone/repository?endpointId=443399" \
+  -d '{
+    "name": "my-new-stack",
+    "repositoryURL": "https://git.vish.gg/Vish/homelab.git",
+    "repositREDACTED_APP_PASSWORD": "refs/heads/main",
+    "composeFile": "hosts/vms/homelab-vm/my-service.yaml",
+    "repositoREDACTED_APP_PASSWORD": true,
+    "reREDACTED_APP_PASSWORD": "",
+    "reREDACTED_APP_PASSWORD": "YOUR_GIT_TOKEN",
+    "autoUpdate": {
+      "interval": "5m",
+      "forceUpdate": false,
+      "forcePullImage": false
+    }
+  }'
+```
+
+**Endpoint IDs:**
+| Endpoint | ID |
+|----------|-----|
+| Atlantis | 2 |
+| Calypso | 443397 |
+| Homelab VM | 443399 |
+| RPi5 | 443395 |
+| Concord NUC | 443398 |
+
+---
+
+## 📊 Monitoring Sync Status
+
+### Check Stack Versions
+
+Each stack shows its current Git commit hash. Compare with the repo:
+
+```bash
+# Get current repo HEAD
+git log -1 --format="%H"
+
+# Check in Portainer
+# Stack → GitConfig → ConfigHash should match
+```
+
+### Common Sync States
+
+| ConfigHash matches HEAD | Stack files changed | Result |
+|------------------------|---------------------|--------|
+| ✅ Yes | N/A | Up to date |
+| ❌ No | ✅ Yes | Will update on next poll |
+| ❌ No | ❌ No | Expected - stack unchanged |
+
+### Troubleshooting Sync Issues
+
+**Stack not updating:**
+1. Check if the specific compose file changed (not just any file)
+2. Verify Git credentials in Portainer are valid
+3. Check Portainer logs for fetch errors
+4. Try manual "Pull and redeploy"
+
+**Wrong version deployed:**
+1. Verify the branch is `refs/heads/main`
+2. Check compose file path matches (watch for symlinks)
+3. Clear Portainer's git cache by recreating the stack
+
+---
+
+## 🔐 Git Authentication
+
+Stacks use a shared Git credential configured in Portainer:
+
+| Setting | Value |
+|---------|-------|
+| **Credential ID** | 1 |
+| **Repository** | `https://git.vish.gg/Vish/homelab.git` |
+| **Auth Type** | Token-based |
+
+To update credentials:
+1. Portainer → Settings → Credentials
+2. Update the Git credential
+3. All stacks using that credential will use the new token
+
+---
+
+## 📋 Best Practices
+
+### Do ✅
+
+- Use descriptive commit messages for stack changes
+- Test compose files locally before pushing
+- Keep one service per compose file when possible
+- Use canonical `hosts/…` paths in Portainer for new stacks (not symlink paths)
+- Enable re-pull for stable `:latest` services
+
+### Don't ❌
+
+- Force redeployment (causes unnecessary restarts)
+- Use `latest` tag for databases
+- Push broken compose files to main
+- Manually edit stacks in Portainer (changes will be overwritten)
+
+---
+
+## 🔗 Related Documentation
+
+- **[Deployment Guide](deployment.md)** - How to create new services
+- **[Monitoring Setup](monitoring.md)** - Track stack health
+- **[Troubleshooting](../troubleshooting/common-issues.md)** - Common problems
+
+---
+
+*Last updated: March 2026*