Sanitized mirror from private repository - 2026-03-30 00:10:29 UTC
This commit is contained in:
Gitea Mirror Bot
301
docs/admin/IMAGE_UPDATE_GUIDE.md
Normal file
301
docs/admin/IMAGE_UPDATE_GUIDE.md
Normal file
@@ -0,0 +1,301 @@
|
||||
# Docker Image Update Strategy
|
||||
|
||||
Last updated: 2026-03-17
|
||||
|
||||
## Overview
|
||||
|
||||
The homelab uses a multi-layered approach to keeping Docker images up to date, combining automated detection, GitOps deployment, and manual controls.
|
||||
|
||||
```
|
||||
Renovate (weekly scan) ──► Creates PR with version bumps
|
||||
│
|
||||
Merge PR to main
|
||||
│
|
||||
portainer-deploy.yml (CI) ──► Redeploys changed stacks (pullImage=true)
|
||||
│
|
||||
Images pulled & containers recreated
|
||||
│
|
||||
DIUN (weekly scan) ──────► Notifies via ntfy if images still outdated
|
||||
│
|
||||
Watchtower (on-demand) ──► Manual trigger for emergency updates
|
||||
```
|
||||
|
||||
## Update Mechanisms
|
||||
|
||||
### 1. Renovate Bot (Recommended — GitOps)
|
||||
|
||||
Renovate scans all compose files weekly and creates PRs to bump image tags.
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| **Schedule** | Mondays 06:00 UTC |
|
||||
| **Workflow** | `.gitea/workflows/renovate.yml` |
|
||||
| **Config** | `renovate.json` |
|
||||
| **Automerge** | No (requires manual review) |
|
||||
| **Minimum age** | 3 days (avoids broken releases) |
|
||||
| **Scope** | All `docker-compose` files in `hosts/` |
|
||||
|
||||
**How it works:**
|
||||
1. Renovate detects new image versions in compose files
|
||||
2. Creates a PR on Gitea (e.g., "Update linuxserver/sonarr to v4.1.2")
|
||||
3. You review and merge the PR
|
||||
4. `portainer-deploy.yml` CI triggers and redeploys the stack with `pullImage: true`
|
||||
5. Portainer pulls the new image and recreates the container
|
||||
|
||||
**Manual trigger:**
|
||||
```bash
|
||||
# Run Renovate on-demand from Gitea UI:
|
||||
# Actions → renovate → Run workflow
|
||||
```
|
||||
|
||||
### 2. Portainer GitOps Auto-Deploy (CI/CD)
|
||||
|
||||
When compose files are pushed to `main`, the CI workflow auto-redeploys affected stacks.
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| **Workflow** | `.gitea/workflows/portainer-deploy.yml` |
|
||||
| **Trigger** | Push to `main` touching `hosts/**` or `common/**` |
|
||||
| **Pull images** | Yes (`pullImage: true` in redeploy request) |
|
||||
| **Endpoints** | Atlantis, Calypso, NUC, Homelab VM, RPi 5 |
|
||||
|
||||
**All stacks across all endpoints are GitOps-linked (as of 2026-03-17).** Every stack has a `GitConfig` pointing to the repo, so any compose file change triggers an automatic redeploy.
|
||||
|
||||
**To update a specific service manually via GitOps:**
|
||||
```bash
|
||||
# Edit the compose file to bump the image tag
|
||||
vim hosts/synology/atlantis/sonarr.yaml
|
||||
# Change: image: linuxserver/sonarr:latest
|
||||
# To: image: linuxserver/sonarr:4.1.2
|
||||
|
||||
# Commit and push
|
||||
git add hosts/synology/atlantis/sonarr.yaml
|
||||
git commit -m "feat: update sonarr to 4.1.2"
|
||||
git push
|
||||
# CI auto-deploys within ~30 seconds
|
||||
```
|
||||
|
||||
### 3. DIUN — Docker Image Update Notifier (Detection)
|
||||
|
||||
DIUN monitors all running containers and sends ntfy notifications when upstream images have new digests.
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| **Host** | Atlantis |
|
||||
| **Schedule** | Mondays 09:00 UTC (3 hours after Renovate) |
|
||||
| **Compose** | `hosts/synology/atlantis/diun.yaml` |
|
||||
| **Notifications** | ntfy topic `diun` (https://ntfy.vish.gg/diun) |
|
||||
|
||||
DIUN is detection-only — it tells you what's outdated but doesn't update anything. If Renovate missed something (e.g., a `:latest` tag with a new digest), DIUN will catch it.
|
||||
|
||||
### 4. Watchtower (On-Demand Manual Updates)
|
||||
|
||||
Watchtower runs on 3 endpoints with automatic updates **disabled**. It's configured for manual HTTP API triggers only.
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| **Hosts** | Atlantis, Calypso, Homelab VM |
|
||||
| **Schedule** | Disabled (manual only) |
|
||||
| **Compose** | `common/watchtower-full.yaml` |
|
||||
| **API port** | 8083 (configurable via `WATCHTOWER_PORT`) |
|
||||
| **Notifications** | ntfy via shoutrrr |
|
||||
|
||||
**Trigger a manual update on a specific host:**
|
||||
```bash
|
||||
# Atlantis
|
||||
curl -X POST http://192.168.0.200:8083/v1/update \
|
||||
-H "Authorization: Bearer watchtower-metrics-token"
|
||||
|
||||
# Calypso
|
||||
curl -X POST http://192.168.0.250:8083/v1/update \
|
||||
-H "Authorization: Bearer watchtower-metrics-token"
|
||||
|
||||
# Homelab VM
|
||||
curl -X POST http://localhost:8083/v1/update \
|
||||
-H "Authorization: Bearer watchtower-metrics-token"
|
||||
```
|
||||
|
||||
This pulls the latest image for every container on that host and recreates any that have newer images. Use sparingly — it updates everything at once.
|
||||
|
||||
**Exclude a container from Watchtower:**
|
||||
```yaml
|
||||
labels:
|
||||
- "com.centurylinklabs.watchtower.enable=false"
|
||||
```
|
||||
|
||||
### 5. Portainer UI (Manual Per-Stack)
|
||||
|
||||
For individual stack updates via the Portainer web UI:
|
||||
|
||||
1. Go to https://192.168.0.200:9443
|
||||
2. Navigate to Stacks → select the stack
|
||||
3. Click **Pull and redeploy** (pulls latest images)
|
||||
4. Or click **Update the stack** → check "Pull latest image"
|
||||
|
||||
## Recommended Workflow
|
||||
|
||||
### Weekly Routine (Automated)
|
||||
|
||||
```
|
||||
Monday 06:00 UTC → Renovate creates PRs for version bumps
|
||||
Monday 09:00 UTC → DIUN sends digest change notifications
|
||||
```
|
||||
|
||||
1. Check ntfy for DIUN notifications and Gitea for Renovate PRs
|
||||
2. Review and merge Renovate PRs (CI auto-deploys)
|
||||
3. For `:latest` tag updates (no version to bump), redeploy the stack via Portainer
|
||||
|
||||
### Updating a Single Service (Step-by-Step)
|
||||
|
||||
**Method 1: Portainer Redeploy (simplest, recommended for `:latest` tags)**
|
||||
|
||||
1. Open Portainer: https://192.168.0.200:9443
|
||||
2. Go to Stacks → select the stack
|
||||
3. Click **Pull and redeploy** (or **Update the stack** → check "Re-pull image")
|
||||
4. Verify the container is healthy after redeploy
|
||||
|
||||
Or via Portainer API:
|
||||
```bash
|
||||
# Redeploy a GitOps stack (pulls latest from git + pulls images)
|
||||
curl -sk -X PUT "https://192.168.0.200:9443/api/stacks/<STACK_ID>/git/redeploy?endpointId=2" \
|
||||
-H "X-API-Key: "REDACTED_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"pullImage": true, "prune": true, "repositoryAuthentication": true, "repositoryUsername": "vish", "repositoryPassword": "<GITEA_TOKEN>"}'
|
||||
```
|
||||
|
||||
Or via MCP (from opencode/Claude Code):
|
||||
```
|
||||
redeploy_stack("sonarr-stack")
|
||||
```
|
||||
|
||||
**Method 2: Git commit (recommended for version-pinned images)**
|
||||
|
||||
```bash
|
||||
# 1. Edit the compose file
|
||||
vim hosts/synology/atlantis/arr-suite/docker-compose.yml
|
||||
# Change: image: linuxserver/sonarr:4.0.0
|
||||
# To: image: linuxserver/sonarr:4.1.2
|
||||
|
||||
# 2. Commit and push
|
||||
git add hosts/synology/atlantis/arr-suite/docker-compose.yml
|
||||
git commit -m "feat: update sonarr to 4.1.2"
|
||||
git push
|
||||
|
||||
# 3. CI auto-deploys within ~30 seconds via portainer-deploy.yml
|
||||
```
|
||||
|
||||
**Method 3: Watchtower (emergency — updates ALL containers on a host)**
|
||||
|
||||
```bash
|
||||
curl -X POST http://192.168.0.200:8083/v1/update \
|
||||
-H "Authorization: Bearer watchtower-metrics-token"
|
||||
```
|
||||
|
||||
Use sparingly — this pulls and recreates every container on the host.
|
||||
|
||||
### Updating All Services on a Host
|
||||
|
||||
```bash
|
||||
# Trigger Watchtower on the host
|
||||
curl -X POST http://<host-ip>:8083/v1/update \
|
||||
-H "Authorization: Bearer watchtower-metrics-token"
|
||||
|
||||
# Or redeploy all stacks via Portainer API
|
||||
# (the portainer-deploy CI does this automatically on git push)
|
||||
```
|
||||
|
||||
### Verifying an Update
|
||||
|
||||
After any update method, verify the container is healthy:
|
||||
|
||||
```bash
|
||||
# Via MCP
|
||||
list_stack_containers("sonarr-stack")
|
||||
check_url("http://192.168.0.200:8989")
|
||||
|
||||
# Via CLI
|
||||
ssh atlantis "/usr/local/bin/docker ps --filter name=sonarr --format '{{.Names}}: {{.Image}} ({{.Status}})'"
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
### Orphan Containers After Manual `docker compose up`
|
||||
|
||||
If you run `docker compose up` directly on a host (not through Portainer), the containers get a different compose project label than the Portainer-managed stack. This creates:
|
||||
|
||||
- A "Limited" ghost entry in the Portainer Stacks UI
|
||||
- Redeploy failures: "container name already in use"
|
||||
|
||||
**Fix:** Stop and remove the orphaned containers, then redeploy via Portainer.
|
||||
|
||||
**Prevention:** Always update through Portainer (UI, API, or GitOps CI). Never run `docker compose up` directly for Portainer-managed stacks.
|
||||
|
||||
### Git Auth Failures on Redeploy
|
||||
|
||||
If a stack redeploy returns "authentication required", the Gitea credentials cached in the stack are stale. Pass the service account token in the redeploy request (see Method 1 above).
|
||||
|
||||
## Image Tagging Strategy
|
||||
|
||||
| Strategy | Used By | Pros | Cons |
|
||||
|----------|---------|------|------|
|
||||
| `:latest` | Most services | Always newest, simple | Can break, no rollback, Renovate can't bump |
|
||||
| `:version` (e.g., `:4.1.2`) | Critical services | Deterministic, Renovate can bump | Requires manual/Renovate updates |
|
||||
| `:major` (e.g., `:4`) | Some LinuxServer images | Auto-updates within major | May get breaking minor changes |
|
||||
|
||||
**Recommendation:** Use specific version tags for critical services (Plex, Sonarr, Radarr, Authentik, Gitea, PostgreSQL). Use `:latest` for non-critical/replaceable services (IT-Tools, theme-park, iperf3).
|
||||
|
||||
## Services That CANNOT Be GitOps Deployed
|
||||
|
||||
These two services are **bootstrap dependencies** for the GitOps pipeline itself. They must be managed manually via `docker compose` or through Portainer UI — never through the CI/CD workflow.
|
||||
|
||||
| Service | Host | Reason |
|
||||
|---------|------|--------|
|
||||
| **Gitea** | Calypso | Hosts the git repository. CI/CD pulls code from Gitea, so auto-deploying Gitea via CI creates a chicken-and-egg problem. If Gitea goes down during a redeploy, the pipeline can't recover. |
|
||||
| **Nginx Proxy Manager** | matrix-ubuntu | Routes all HTTPS traffic including `git.vish.gg`. Removing NPM to recreate it as a GitOps stack kills access to Gitea, which prevents the GitOps stack from being created. |
|
||||
|
||||
**To update these manually:**
|
||||
```bash
|
||||
# Gitea
|
||||
ssh calypso
|
||||
cd /volume1/docker/gitea
|
||||
sudo /var/packages/REDACTED_APP_PASSWORD/target/usr/bin/docker compose pull
|
||||
sudo /var/packages/REDACTED_APP_PASSWORD/target/usr/bin/docker compose up -d
|
||||
|
||||
# Nginx Proxy Manager
|
||||
ssh matrix-ubuntu
|
||||
cd /opt/npm
|
||||
sudo docker compose pull
|
||||
sudo docker compose up -d
|
||||
```
|
||||
|
||||
## Services NOT Auto-Updated
|
||||
|
||||
These services should be updated manually with care:
|
||||
|
||||
| Service | Reason |
|
||||
|---------|--------|
|
||||
| **Gitea** | Bootstrap dependency (see above) |
|
||||
| **Nginx Proxy Manager** | Bootstrap dependency on matrix-ubuntu (see above) |
|
||||
| **Authentik** | SSO provider — broken update locks out all services |
|
||||
| **PostgreSQL** | Database — major version upgrades require migration |
|
||||
| **Portainer** | Container orchestrator — update via DSM or manual Docker commands |
|
||||
|
||||
## Monitoring Update Status
|
||||
|
||||
```bash
|
||||
# Check which images are outdated (via DIUN ntfy topic)
|
||||
# Subscribe to: https://ntfy.vish.gg/diun
|
||||
|
||||
# Check Watchtower metrics
|
||||
curl http://192.168.0.200:8083/v1/metrics \
|
||||
-H "Authorization: Bearer watchtower-metrics-token"
|
||||
|
||||
# Check running image digests vs remote
|
||||
docker images --digests | grep <image-name>
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Ansible Playbook Guide](ANSIBLE_PLAYBOOK_GUIDE.md) — System package updates
|
||||
- [Portainer API Guide](PORTAINER_API_GUIDE.md) — Stack management API
|
||||
- [GitOps Guide](gitops.md) — CI/CD pipeline details
|
||||
Reference in New Issue
Block a user