Sanitized mirror from private repository - 2026-03-22 08:40:48 UTC
This commit is contained in:
Gitea Mirror Bot
241
docs/admin/IMAGE_UPDATE_GUIDE.md
Normal file
241
docs/admin/IMAGE_UPDATE_GUIDE.md
Normal file
@@ -0,0 +1,241 @@
|
||||
# Docker Image Update Strategy
|
||||
|
||||
Last updated: 2026-03-17
|
||||
|
||||
## Overview
|
||||
|
||||
The homelab uses a multi-layered approach to keeping Docker images up to date, combining automated detection, GitOps deployment, and manual controls.
|
||||
|
||||
```
|
||||
Renovate (weekly scan) ──► Creates PR with version bumps
|
||||
│
|
||||
Merge PR to main
|
||||
│
|
||||
portainer-deploy.yml (CI) ──► Redeploys changed stacks (pullImage=true)
|
||||
│
|
||||
Images pulled & containers recreated
|
||||
│
|
||||
DIUN (weekly scan) ──────► Notifies via ntfy if images still outdated
|
||||
│
|
||||
Watchtower (on-demand) ──► Manual trigger for emergency updates
|
||||
```
|
||||
|
||||
## Update Mechanisms
|
||||
|
||||
### 1. Renovate Bot (Recommended — GitOps)
|
||||
|
||||
Renovate scans all compose files weekly and creates PRs to bump image tags.
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| **Schedule** | Mondays 06:00 UTC |
|
||||
| **Workflow** | `.gitea/workflows/renovate.yml` |
|
||||
| **Config** | `renovate.json` |
|
||||
| **Automerge** | No (requires manual review) |
|
||||
| **Minimum age** | 3 days (avoids broken releases) |
|
||||
| **Scope** | All `docker-compose` files in `hosts/` |
|
||||
|
||||
**How it works:**
|
||||
1. Renovate detects new image versions in compose files
|
||||
2. Creates a PR on Gitea (e.g., "Update linuxserver/sonarr to v4.1.2")
|
||||
3. You review and merge the PR
|
||||
4. `portainer-deploy.yml` CI triggers and redeploys the stack with `pullImage: true`
|
||||
5. Portainer pulls the new image and recreates the container
|
||||
|
||||
**Manual trigger:**
|
||||
```bash
|
||||
# Run Renovate on-demand from Gitea UI:
|
||||
# Actions → renovate → Run workflow
|
||||
```
|
||||
|
||||
### 2. Portainer GitOps Auto-Deploy (CI/CD)
|
||||
|
||||
When compose files are pushed to `main`, the CI workflow auto-redeploys affected stacks.
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| **Workflow** | `.gitea/workflows/portainer-deploy.yml` |
|
||||
| **Trigger** | Push to `main` touching `hosts/**` or `common/**` |
|
||||
| **Pull images** | Yes (`pullImage: true` in redeploy request) |
|
||||
| **Endpoints** | Atlantis, Calypso, NUC, Homelab VM, RPi 5 |
|
||||
|
||||
**All stacks across all endpoints are GitOps-linked (as of 2026-03-17).** Every stack has a `GitConfig` pointing to the repo, so any compose file change triggers an automatic redeploy.
|
||||
|
||||
**To update a specific service manually via GitOps:**
|
||||
```bash
|
||||
# Edit the compose file to bump the image tag
|
||||
vim hosts/synology/atlantis/sonarr.yaml
|
||||
# Change: image: linuxserver/sonarr:latest
|
||||
# To: image: linuxserver/sonarr:4.1.2
|
||||
|
||||
# Commit and push
|
||||
git add hosts/synology/atlantis/sonarr.yaml
|
||||
git commit -m "feat: update sonarr to 4.1.2"
|
||||
git push
|
||||
# CI auto-deploys within ~30 seconds
|
||||
```
|
||||
|
||||
### 3. DIUN — Docker Image Update Notifier (Detection)
|
||||
|
||||
DIUN monitors all running containers and sends ntfy notifications when upstream images have new digests.
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| **Host** | Atlantis |
|
||||
| **Schedule** | Mondays 09:00 UTC (3 hours after Renovate) |
|
||||
| **Compose** | `hosts/synology/atlantis/diun.yaml` |
|
||||
| **Notifications** | ntfy topic `diun` (https://ntfy.vish.gg/diun) |
|
||||
|
||||
DIUN is detection-only — it tells you what's outdated but doesn't update anything. If Renovate missed something (e.g., a `:latest` tag with a new digest), DIUN will catch it.
|
||||
|
||||
### 4. Watchtower (On-Demand Manual Updates)
|
||||
|
||||
Watchtower runs on 3 endpoints with automatic updates **disabled**. It's configured for manual HTTP API triggers only.
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| **Hosts** | Atlantis, Calypso, Homelab VM |
|
||||
| **Schedule** | Disabled (manual only) |
|
||||
| **Compose** | `common/watchtower-full.yaml` |
|
||||
| **API port** | 8083 (configurable via `WATCHTOWER_PORT`) |
|
||||
| **Notifications** | ntfy via shoutrrr |
|
||||
|
||||
**Trigger a manual update on a specific host:**
|
||||
```bash
|
||||
# Atlantis
|
||||
curl -X POST http://192.168.0.200:8083/v1/update \
|
||||
-H "Authorization: Bearer watchtower-metrics-token"
|
||||
|
||||
# Calypso
|
||||
curl -X POST http://192.168.0.250:8083/v1/update \
|
||||
-H "Authorization: Bearer watchtower-metrics-token"
|
||||
|
||||
# Homelab VM
|
||||
curl -X POST http://localhost:8083/v1/update \
|
||||
-H "Authorization: Bearer watchtower-metrics-token"
|
||||
```
|
||||
|
||||
This pulls the latest image for every container on that host and recreates any that have newer images. Use sparingly — it updates everything at once.
|
||||
|
||||
**Exclude a container from Watchtower:**
|
||||
```yaml
|
||||
labels:
|
||||
- "com.centurylinklabs.watchtower.enable=false"
|
||||
```
|
||||
|
||||
### 5. Portainer UI (Manual Per-Stack)
|
||||
|
||||
For individual stack updates via the Portainer web UI:
|
||||
|
||||
1. Go to https://192.168.0.200:9443
|
||||
2. Navigate to Stacks → select the stack
|
||||
3. Click **Pull and redeploy** (pulls latest images)
|
||||
4. Or click **Update the stack** → check "Pull latest image"
|
||||
|
||||
## Recommended Workflow
|
||||
|
||||
### Weekly Routine (Automated)
|
||||
|
||||
```
|
||||
Monday 06:00 UTC → Renovate creates PRs for version bumps
|
||||
Monday 09:00 UTC → DIUN sends digest change notifications
|
||||
```
|
||||
|
||||
1. Check ntfy for DIUN notifications and Gitea for Renovate PRs
|
||||
2. Review and merge Renovate PRs (CI auto-deploys)
|
||||
3. For `:latest` tag updates (no version to bump), trigger Watchtower manually
|
||||
|
||||
### Updating a Single Service
|
||||
|
||||
```bash
|
||||
# Option A: Via Portainer UI
|
||||
# Stacks → stack-name → Pull and redeploy
|
||||
|
||||
# Option B: Via git (recommended for traceability)
|
||||
# Edit compose, commit, push → CI deploys
|
||||
|
||||
# Option C: Via Watchtower API (updates all containers on that host)
|
||||
curl -X POST http://<host>:8083/v1/update \
|
||||
-H "Authorization: Bearer watchtower-metrics-token"
|
||||
|
||||
# Option D: Via Ansible (for system packages, not Docker)
|
||||
ansible-playbook -i ansible/inventory.yml \
|
||||
ansible/automation/playbooks/update_system.yml --limit <host>
|
||||
```
|
||||
|
||||
### Updating All Services on a Host
|
||||
|
||||
```bash
|
||||
# Trigger Watchtower on the host
|
||||
curl -X POST http://<host-ip>:8083/v1/update \
|
||||
-H "Authorization: Bearer watchtower-metrics-token"
|
||||
|
||||
# Or redeploy all stacks via Portainer API
|
||||
# (the portainer-deploy CI does this automatically on git push)
|
||||
```
|
||||
|
||||
## Image Tagging Strategy
|
||||
|
||||
| Strategy | Used By | Pros | Cons |
|
||||
|----------|---------|------|------|
|
||||
| `:latest` | Most services | Always newest, simple | Can break, no rollback, Renovate can't bump |
|
||||
| `:version` (e.g., `:4.1.2`) | Critical services | Deterministic, Renovate can bump | Requires manual/Renovate updates |
|
||||
| `:major` (e.g., `:4`) | Some LinuxServer images | Auto-updates within major | May get breaking minor changes |
|
||||
|
||||
**Recommendation:** Use specific version tags for critical services (Plex, Sonarr, Radarr, Authentik, Gitea, PostgreSQL). Use `:latest` for non-critical/replaceable services (IT-Tools, theme-park, iperf3).
|
||||
|
||||
## Services That CANNOT Be GitOps Deployed
|
||||
|
||||
These two services are **bootstrap dependencies** for the GitOps pipeline itself. They must be managed manually via `docker compose` or through Portainer UI — never through the CI/CD workflow.
|
||||
|
||||
| Service | Host | Reason |
|
||||
|---------|------|--------|
|
||||
| **Gitea** | Calypso | Hosts the git repository. CI/CD pulls code from Gitea, so auto-deploying Gitea via CI creates a chicken-and-egg problem. If Gitea goes down during a redeploy, the pipeline can't recover. |
|
||||
| **Nginx Proxy Manager** | matrix-ubuntu | Routes all HTTPS traffic including `git.vish.gg`. Removing NPM to recreate it as a GitOps stack kills access to Gitea, which prevents the GitOps stack from being created. |
|
||||
|
||||
**To update these manually:**
|
||||
```bash
|
||||
# Gitea
|
||||
ssh calypso
|
||||
cd /volume1/docker/gitea
|
||||
sudo /var/packages/REDACTED_APP_PASSWORD/target/usr/bin/docker compose pull
|
||||
sudo /var/packages/REDACTED_APP_PASSWORD/target/usr/bin/docker compose up -d
|
||||
|
||||
# Nginx Proxy Manager
|
||||
ssh matrix-ubuntu
|
||||
cd /opt/npm
|
||||
sudo docker compose pull
|
||||
sudo docker compose up -d
|
||||
```
|
||||
|
||||
## Services NOT Auto-Updated
|
||||
|
||||
These services should be updated manually with care:
|
||||
|
||||
| Service | Reason |
|
||||
|---------|--------|
|
||||
| **Gitea** | Bootstrap dependency (see above) |
|
||||
| **Nginx Proxy Manager** | Bootstrap dependency on matrix-ubuntu (see above) |
|
||||
| **Authentik** | SSO provider — broken update locks out all services |
|
||||
| **PostgreSQL** | Database — major version upgrades require migration |
|
||||
| **Portainer** | Container orchestrator — update via DSM or manual Docker commands |
|
||||
|
||||
## Monitoring Update Status
|
||||
|
||||
```bash
|
||||
# Check which images are outdated (via DIUN ntfy topic)
|
||||
# Subscribe to: https://ntfy.vish.gg/diun
|
||||
|
||||
# Check Watchtower metrics
|
||||
curl http://192.168.0.200:8083/v1/metrics \
|
||||
-H "Authorization: Bearer watchtower-metrics-token"
|
||||
|
||||
# Check running image digests vs remote
|
||||
docker images --digests | grep <image-name>
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Ansible Playbook Guide](ANSIBLE_PLAYBOOK_GUIDE.md) — System package updates
|
||||
- [Portainer API Guide](PORTAINER_API_GUIDE.md) — Stack management API
|
||||
- [GitOps Guide](gitops.md) — CI/CD pipeline details
|
||||
Reference in New Issue
Block a user