Sanitized mirror from private repository - 2026-04-05 08:31:50 UTC

2026-04-05 08:31:50 +00:00
commit 2be8f1fe17
1390 changed files with 354479 additions and 0 deletions
--- a/docs/admin/IMAGE_UPDATE_GUIDE.md
+++ b/docs/admin/IMAGE_UPDATE_GUIDE.md
@@ -0,0 +1,301 @@
+# Docker Image Update Strategy
+
+Last updated: 2026-03-17
+
+## Overview
+
+The homelab uses a multi-layered approach to keeping Docker images up to date, combining automated detection, GitOps deployment, and manual controls.
+
+```
+Renovate (weekly scan) ──► Creates PR with version bumps
+                                    │
+                              Merge PR to main
+                                    │
+portainer-deploy.yml (CI) ──► Redeploys changed stacks (pullImage=true)
+                                    │
+                              Images pulled & containers recreated
+                                    │
+DIUN (weekly scan) ──────► Notifies via ntfy if images still outdated
+                                    │
+Watchtower (on-demand) ──► Manual trigger for emergency updates
+```
+
+## Update Mechanisms
+
+### 1. Renovate Bot (Recommended — GitOps)
+
+Renovate scans all compose files weekly and creates PRs to bump image tags.
+
+| Setting | Value |
+|---------|-------|
+| **Schedule** | Mondays 06:00 UTC |
+| **Workflow** | `.gitea/workflows/renovate.yml` |
+| **Config** | `renovate.json` |
+| **Automerge** | No (requires manual review) |
+| **Minimum age** | 3 days (avoids broken releases) |
+| **Scope** | All `docker-compose` files in `hosts/` |
+
+**How it works:**
+1. Renovate detects new image versions in compose files
+2. Creates a PR on Gitea (e.g., "Update linuxserver/sonarr to v4.1.2")
+3. You review and merge the PR
+4. `portainer-deploy.yml` CI triggers and redeploys the stack with `pullImage: true`
+5. Portainer pulls the new image and recreates the container
+
+**Manual trigger:**
+```bash
+# Run Renovate on-demand from Gitea UI:
+# Actions → renovate → Run workflow
+```
+
+### 2. Portainer GitOps Auto-Deploy (CI/CD)
+
+When compose files are pushed to `main`, the CI workflow auto-redeploys affected stacks.
+
+| Setting | Value |
+|---------|-------|
+| **Workflow** | `.gitea/workflows/portainer-deploy.yml` |
+| **Trigger** | Push to `main` touching `hosts/**` or `common/**` |
+| **Pull images** | Yes (`pullImage: true` in redeploy request) |
+| **Endpoints** | Atlantis, Calypso, NUC, Homelab VM, RPi 5 |
+
+**All stacks across all endpoints are GitOps-linked (as of 2026-03-17).** Every stack has a `GitConfig` pointing to the repo, so any compose file change triggers an automatic redeploy.
+
+**To update a specific service manually via GitOps:**
+```bash
+# Edit the compose file to bump the image tag
+vim hosts/synology/atlantis/sonarr.yaml
+# Change: image: linuxserver/sonarr:latest
+# To:     image: linuxserver/sonarr:4.1.2
+
+# Commit and push
+git add hosts/synology/atlantis/sonarr.yaml
+git commit -m "feat: update sonarr to 4.1.2"
+git push
+# CI auto-deploys within ~30 seconds
+```
+
+### 3. DIUN — Docker Image Update Notifier (Detection)
+
+DIUN monitors all running containers and sends ntfy notifications when upstream images have new digests.
+
+| Setting | Value |
+|---------|-------|
+| **Host** | Atlantis |
+| **Schedule** | Mondays 09:00 UTC (3 hours after Renovate) |
+| **Compose** | `hosts/synology/atlantis/diun.yaml` |
+| **Notifications** | ntfy topic `diun` (https://ntfy.vish.gg/diun) |
+
+DIUN is detection-only — it tells you what's outdated but doesn't update anything. If Renovate missed something (e.g., a `:latest` tag with a new digest), DIUN will catch it.
+
+### 4. Watchtower (On-Demand Manual Updates)
+
+Watchtower runs on 3 endpoints with automatic updates **disabled**. It's configured for manual HTTP API triggers only.
+
+| Setting | Value |
+|---------|-------|
+| **Hosts** | Atlantis, Calypso, Homelab VM |
+| **Schedule** | Disabled (manual only) |
+| **Compose** | `common/watchtower-full.yaml` |
+| **API port** | 8083 (configurable via `WATCHTOWER_PORT`) |
+| **Notifications** | ntfy via shoutrrr |
+
+**Trigger a manual update on a specific host:**
+```bash
+# Atlantis
+curl -X POST http://192.168.0.200:8083/v1/update \
+  -H "Authorization: Bearer watchtower-metrics-token"
+
+# Calypso
+curl -X POST http://192.168.0.250:8083/v1/update \
+  -H "Authorization: Bearer watchtower-metrics-token"
+
+# Homelab VM
+curl -X POST http://localhost:8083/v1/update \
+  -H "Authorization: Bearer watchtower-metrics-token"
+```
+
+This pulls the latest image for every container on that host and recreates any that have newer images. Use sparingly — it updates everything at once.
+
+**Exclude a container from Watchtower:**
+```yaml
+labels:
+  - "com.centurylinklabs.watchtower.enable=false"
+```
+
+### 5. Portainer UI (Manual Per-Stack)
+
+For individual stack updates via the Portainer web UI:
+
+1. Go to https://192.168.0.200:9443
+2. Navigate to Stacks → select the stack
+3. Click **Pull and redeploy** (pulls latest images)
+4. Or click **Update the stack** → check "Pull latest image"
+
+## Recommended Workflow
+
+### Weekly Routine (Automated)
+
+```
+Monday 06:00 UTC  →  Renovate creates PRs for version bumps
+Monday 09:00 UTC  →  DIUN sends digest change notifications
+```
+
+1. Check ntfy for DIUN notifications and Gitea for Renovate PRs
+2. Review and merge Renovate PRs (CI auto-deploys)
+3. For `:latest` tag updates (no version to bump), redeploy the stack via Portainer
+
+### Updating a Single Service (Step-by-Step)
+
+**Method 1: Portainer Redeploy (simplest, recommended for `:latest` tags)**
+
+1. Open Portainer: https://192.168.0.200:9443
+2. Go to Stacks → select the stack
+3. Click **Pull and redeploy** (or **Update the stack** → check "Re-pull image")
+4. Verify the container is healthy after redeploy
+
+Or via Portainer API:
+```bash
+# Redeploy a GitOps stack (pulls latest from git + pulls images)
+curl -sk -X PUT "https://192.168.0.200:9443/api/stacks/<STACK_ID>/git/redeploy?endpointId=2" \
+  -H "X-API-Key: "REDACTED_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"pullImage": true, "prune": true, "repositoryAuthentication": true, "repositoryUsername": "vish", "repositoryPassword": "<GITEA_TOKEN>"}'
+```
+
+Or via MCP (from opencode/Claude Code):
+```
+redeploy_stack("sonarr-stack")
+```
+
+**Method 2: Git commit (recommended for version-pinned images)**
+
+```bash
+# 1. Edit the compose file
+vim hosts/synology/atlantis/arr-suite/docker-compose.yml
+# Change: image: linuxserver/sonarr:4.0.0
+# To:     image: linuxserver/sonarr:4.1.2
+
+# 2. Commit and push
+git add hosts/synology/atlantis/arr-suite/docker-compose.yml
+git commit -m "feat: update sonarr to 4.1.2"
+git push
+
+# 3. CI auto-deploys within ~30 seconds via portainer-deploy.yml
+```
+
+**Method 3: Watchtower (emergency — updates ALL containers on a host)**
+
+```bash
+curl -X POST http://192.168.0.200:8083/v1/update \
+  -H "Authorization: Bearer watchtower-metrics-token"
+```
+
+Use sparingly — this pulls and recreates every container on the host.
+
+### Updating All Services on a Host
+
+```bash
+# Trigger Watchtower on the host
+curl -X POST http://<host-ip>:8083/v1/update \
+  -H "Authorization: Bearer watchtower-metrics-token"
+
+# Or redeploy all stacks via Portainer API
+# (the portainer-deploy CI does this automatically on git push)
+```
+
+### Verifying an Update
+
+After any update method, verify the container is healthy:
+
+```bash
+# Via MCP
+list_stack_containers("sonarr-stack")
+check_url("http://192.168.0.200:8989")
+
+# Via CLI
+ssh atlantis "/usr/local/bin/docker ps --filter name=sonarr --format '{{.Names}}: {{.Image}} ({{.Status}})'"
+```
+
+## Gotchas
+
+### Orphan Containers After Manual `docker compose up`
+
+If you run `docker compose up` directly on a host (not through Portainer), the containers get a different compose project label than the Portainer-managed stack. This creates:
+
+- A "Limited" ghost entry in the Portainer Stacks UI
+- Redeploy failures: "container name already in use"
+
+**Fix:** Stop and remove the orphaned containers, then redeploy via Portainer.
+
+**Prevention:** Always update through Portainer (UI, API, or GitOps CI). Never run `docker compose up` directly for Portainer-managed stacks.
+
+### Git Auth Failures on Redeploy
+
+If a stack redeploy returns "authentication required", the Gitea credentials cached in the stack are stale. Pass the service account token in the redeploy request (see Method 1 above).
+
+## Image Tagging Strategy
+
+| Strategy | Used By | Pros | Cons |
+|----------|---------|------|------|
+| `:latest` | Most services | Always newest, simple | Can break, no rollback, Renovate can't bump |
+| `:version` (e.g., `:4.1.2`) | Critical services | Deterministic, Renovate can bump | Requires manual/Renovate updates |
+| `:major` (e.g., `:4`) | Some LinuxServer images | Auto-updates within major | May get breaking minor changes |
+
+**Recommendation:** Use specific version tags for critical services (Plex, Sonarr, Radarr, Authentik, Gitea, PostgreSQL). Use `:latest` for non-critical/replaceable services (IT-Tools, theme-park, iperf3).
+
+## Services That CANNOT Be GitOps Deployed
+
+These two services are **bootstrap dependencies** for the GitOps pipeline itself. They must be managed manually via `docker compose` or through Portainer UI — never through the CI/CD workflow.
+
+| Service | Host | Reason |
+|---------|------|--------|
+| **Gitea** | Calypso | Hosts the git repository. CI/CD pulls code from Gitea, so auto-deploying Gitea via CI creates a chicken-and-egg problem. If Gitea goes down during a redeploy, the pipeline can't recover. |
+| **Nginx Proxy Manager** | matrix-ubuntu | Routes all HTTPS traffic including `git.vish.gg`. Removing NPM to recreate it as a GitOps stack kills access to Gitea, which prevents the GitOps stack from being created. |
+
+**To update these manually:**
+```bash
+# Gitea
+ssh calypso
+cd /volume1/docker/gitea
+sudo /var/packages/REDACTED_APP_PASSWORD/target/usr/bin/docker compose pull
+sudo /var/packages/REDACTED_APP_PASSWORD/target/usr/bin/docker compose up -d
+
+# Nginx Proxy Manager
+ssh matrix-ubuntu
+cd /opt/npm
+sudo docker compose pull
+sudo docker compose up -d
+```
+
+## Services NOT Auto-Updated
+
+These services should be updated manually with care:
+
+| Service | Reason |
+|---------|--------|
+| **Gitea** | Bootstrap dependency (see above) |
+| **Nginx Proxy Manager** | Bootstrap dependency on matrix-ubuntu (see above) |
+| **Authentik** | SSO provider — broken update locks out all services |
+| **PostgreSQL** | Database — major version upgrades require migration |
+| **Portainer** | Container orchestrator — update via DSM or manual Docker commands |
+
+## Monitoring Update Status
+
+```bash
+# Check which images are outdated (via DIUN ntfy topic)
+# Subscribe to: https://ntfy.vish.gg/diun
+
+# Check Watchtower metrics
+curl http://192.168.0.200:8083/v1/metrics \
+  -H "Authorization: Bearer watchtower-metrics-token"
+
+# Check running image digests vs remote
+docker images --digests | grep <image-name>
+```
+
+## Related Documentation
+
+- [Ansible Playbook Guide](ANSIBLE_PLAYBOOK_GUIDE.md) — System package updates
+- [Portainer API Guide](PORTAINER_API_GUIDE.md) — Stack management API
+- [GitOps Guide](gitops.md) — CI/CD pipeline details