# Docker Image Update Strategy

Last updated: 2026-03-17

## Overview

The homelab uses a multi-layered approach to keeping Docker images up to date, combining automated detection, GitOps deployment, and manual controls.

```
Renovate (weekly scan) ──► Creates PR with version bumps
                                    │
                              Merge PR to main
                                    │
portainer-deploy.yml (CI) ──► Redeploys changed stacks (pullImage=true)
                                    │
                              Images pulled & containers recreated
                                    │
DIUN (weekly scan) ──────► Notifies via ntfy if images still outdated
                                    │
Watchtower (on-demand) ──► Manual trigger for emergency updates
```

## Update Mechanisms

### 1. Renovate Bot (Recommended — GitOps)

Renovate scans all compose files weekly and creates PRs to bump image tags.

| Setting | Value |
|---------|-------|
| **Schedule** | Mondays 06:00 UTC |
| **Workflow** | `.gitea/workflows/renovate.yml` |
| **Config** | `renovate.json` |
| **Automerge** | No (requires manual review) |
| **Minimum age** | 3 days (avoids broken releases) |
| **Scope** | All `docker-compose` files in `hosts/` |

**How it works:**
1. Renovate detects new image versions in compose files
2. Creates a PR on Gitea (e.g., "Update linuxserver/sonarr to v4.1.2")
3. You review and merge the PR
4. `portainer-deploy.yml` CI triggers and redeploys the stack with `pullImage: true`
5. Portainer pulls the new image and recreates the container

**Manual trigger:**
```bash
# Run Renovate on-demand from Gitea UI:
# Actions → renovate → Run workflow
```

### 2. Portainer GitOps Auto-Deploy (CI/CD)

When compose files are pushed to `main`, the CI workflow auto-redeploys affected stacks.

| Setting | Value |
|---------|-------|
| **Workflow** | `.gitea/workflows/portainer-deploy.yml` |
| **Trigger** | Push to `main` touching `hosts/**` or `common/**` |
| **Pull images** | Yes (`pullImage: true` in redeploy request) |
| **Endpoints** | Atlantis, Calypso, NUC, Homelab VM, RPi 5 |

**All stacks across all endpoints are GitOps-linked (as of 2026-03-17).** Every stack has a `GitConfig` pointing to the repo, so any compose file change triggers an automatic redeploy.

**To update a specific service manually via GitOps:**
```bash
# Edit the compose file to bump the image tag
vim hosts/synology/atlantis/sonarr.yaml
# Change: image: linuxserver/sonarr:latest
# To:     image: linuxserver/sonarr:4.1.2

# Commit and push
git add hosts/synology/atlantis/sonarr.yaml
git commit -m "feat: update sonarr to 4.1.2"
git push
# CI auto-deploys within ~30 seconds
```

### 3. DIUN — Docker Image Update Notifier (Detection)

DIUN monitors all running containers and sends ntfy notifications when upstream images have new digests.

| Setting | Value |
|---------|-------|
| **Host** | Atlantis |
| **Schedule** | Mondays 09:00 UTC (3 hours after Renovate) |
| **Compose** | `hosts/synology/atlantis/diun.yaml` |
| **Notifications** | ntfy topic `diun` (https://ntfy.vish.gg/diun) |

DIUN is detection-only — it tells you what's outdated but doesn't update anything. If Renovate missed something (e.g., a `:latest` tag with a new digest), DIUN will catch it.

### 4. Watchtower (On-Demand Manual Updates)

Watchtower runs on 3 endpoints with automatic updates **disabled**. It's configured for manual HTTP API triggers only.

| Setting | Value |
|---------|-------|
| **Hosts** | Atlantis, Calypso, Homelab VM |
| **Schedule** | Disabled (manual only) |
| **Compose** | `common/watchtower-full.yaml` |
| **API port** | 8083 (configurable via `WATCHTOWER_PORT`) |
| **Notifications** | ntfy via shoutrrr |

**Trigger a manual update on a specific host:**
```bash
# Atlantis
curl -X POST http://192.168.0.200:8083/v1/update \
  -H "Authorization: Bearer watchtower-metrics-token"

# Calypso
curl -X POST http://192.168.0.250:8083/v1/update \
  -H "Authorization: Bearer watchtower-metrics-token"

# Homelab VM
curl -X POST http://localhost:8083/v1/update \
  -H "Authorization: Bearer watchtower-metrics-token"
```

This pulls the latest image for every container on that host and recreates any that have newer images. Use sparingly — it updates everything at once.

**Exclude a container from Watchtower:**
```yaml
labels:
  - "com.centurylinklabs.watchtower.enable=false"
```

### 5. Portainer UI (Manual Per-Stack)

For individual stack updates via the Portainer web UI:

1. Go to https://192.168.0.200:9443
2. Navigate to Stacks → select the stack
3. Click **Pull and redeploy** (pulls latest images)
4. Or click **Update the stack** → check "Pull latest image"

## Recommended Workflow

### Weekly Routine (Automated)

```
Monday 06:00 UTC  →  Renovate creates PRs for version bumps
Monday 09:00 UTC  →  DIUN sends digest change notifications
```

1. Check ntfy for DIUN notifications and Gitea for Renovate PRs
2. Review and merge Renovate PRs (CI auto-deploys)
3. For `:latest` tag updates (no version to bump), redeploy the stack via Portainer

### Updating a Single Service (Step-by-Step)

**Method 1: Portainer Redeploy (simplest, recommended for `:latest` tags)**

1. Open Portainer: https://192.168.0.200:9443
2. Go to Stacks → select the stack
3. Click **Pull and redeploy** (or **Update the stack** → check "Re-pull image")
4. Verify the container is healthy after redeploy

Or via Portainer API:
```bash
# Redeploy a GitOps stack (pulls latest from git + pulls images)
curl -sk -X PUT "https://192.168.0.200:9443/api/stacks/<STACK_ID>/git/redeploy?endpointId=2" \
  -H "X-API-Key: "REDACTED_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"pullImage": true, "prune": true, "repositoryAuthentication": true, "repositoryUsername": "vish", "repositoryPassword": "<GITEA_TOKEN>"}'
```

Or via MCP (from opencode/Claude Code):
```
redeploy_stack("sonarr-stack")
```

**Method 2: Git commit (recommended for version-pinned images)**

```bash
# 1. Edit the compose file
vim hosts/synology/atlantis/arr-suite/docker-compose.yml
# Change: image: linuxserver/sonarr:4.0.0
# To:     image: linuxserver/sonarr:4.1.2

# 2. Commit and push
git add hosts/synology/atlantis/arr-suite/docker-compose.yml
git commit -m "feat: update sonarr to 4.1.2"
git push

# 3. CI auto-deploys within ~30 seconds via portainer-deploy.yml
```

**Method 3: Watchtower (emergency — updates ALL containers on a host)**

```bash
curl -X POST http://192.168.0.200:8083/v1/update \
  -H "Authorization: Bearer watchtower-metrics-token"
```

Use sparingly — this pulls and recreates every container on the host.

### Updating All Services on a Host

```bash
# Trigger Watchtower on the host
curl -X POST http://<host-ip>:8083/v1/update \
  -H "Authorization: Bearer watchtower-metrics-token"

# Or redeploy all stacks via Portainer API
# (the portainer-deploy CI does this automatically on git push)
```

### Verifying an Update

After any update method, verify the container is healthy:

```bash
# Via MCP
list_stack_containers("sonarr-stack")
check_url("http://192.168.0.200:8989")

# Via CLI
ssh atlantis "/usr/local/bin/docker ps --filter name=sonarr --format '{{.Names}}: {{.Image}} ({{.Status}})'"
```

## Gotchas

### Orphan Containers After Manual `docker compose up`

If you run `docker compose up` directly on a host (not through Portainer), the containers get a different compose project label than the Portainer-managed stack. This creates:

- A "Limited" ghost entry in the Portainer Stacks UI
- Redeploy failures: "container name already in use"

**Fix:** Stop and remove the orphaned containers, then redeploy via Portainer.

**Prevention:** Always update through Portainer (UI, API, or GitOps CI). Never run `docker compose up` directly for Portainer-managed stacks.

### Git Auth Failures on Redeploy

If a stack redeploy returns "authentication required", the Gitea credentials cached in the stack are stale. Pass the service account token in the redeploy request (see Method 1 above).

## Image Tagging Strategy

| Strategy | Used By | Pros | Cons |
|----------|---------|------|------|
| `:latest` | Most services | Always newest, simple | Can break, no rollback, Renovate can't bump |
| `:version` (e.g., `:4.1.2`) | Critical services | Deterministic, Renovate can bump | Requires manual/Renovate updates |
| `:major` (e.g., `:4`) | Some LinuxServer images | Auto-updates within major | May get breaking minor changes |

**Recommendation:** Use specific version tags for critical services (Plex, Sonarr, Radarr, Authentik, Gitea, PostgreSQL). Use `:latest` for non-critical/replaceable services (IT-Tools, theme-park, iperf3).

## Services That CANNOT Be GitOps Deployed

These two services are **bootstrap dependencies** for the GitOps pipeline itself. They must be managed manually via `docker compose` or through Portainer UI — never through the CI/CD workflow.

| Service | Host | Reason |
|---------|------|--------|
| **Gitea** | Calypso | Hosts the git repository. CI/CD pulls code from Gitea, so auto-deploying Gitea via CI creates a chicken-and-egg problem. If Gitea goes down during a redeploy, the pipeline can't recover. |
| **Nginx Proxy Manager** | matrix-ubuntu | Routes all HTTPS traffic including `git.vish.gg`. Removing NPM to recreate it as a GitOps stack kills access to Gitea, which prevents the GitOps stack from being created. |

**To update these manually:**
```bash
# Gitea
ssh calypso
cd /volume1/docker/gitea
sudo /var/packages/REDACTED_APP_PASSWORD/target/usr/bin/docker compose pull
sudo /var/packages/REDACTED_APP_PASSWORD/target/usr/bin/docker compose up -d

# Nginx Proxy Manager
ssh matrix-ubuntu
cd /opt/npm
sudo docker compose pull
sudo docker compose up -d
```

## Services NOT Auto-Updated

These services should be updated manually with care:

| Service | Reason |
|---------|--------|
| **Gitea** | Bootstrap dependency (see above) |
| **Nginx Proxy Manager** | Bootstrap dependency on matrix-ubuntu (see above) |
| **Authentik** | SSO provider — broken update locks out all services |
| **PostgreSQL** | Database — major version upgrades require migration |
| **Portainer** | Container orchestrator — update via DSM or manual Docker commands |

## Monitoring Update Status

```bash
# Check which images are outdated (via DIUN ntfy topic)
# Subscribe to: https://ntfy.vish.gg/diun

# Check Watchtower metrics
curl http://192.168.0.200:8083/v1/metrics \
  -H "Authorization: Bearer watchtower-metrics-token"

# Check running image digests vs remote
docker images --digests | grep <image-name>
```

## Related Documentation

- [Ansible Playbook Guide](ANSIBLE_PLAYBOOK_GUIDE.md) — System package updates
- [Portainer API Guide](PORTAINER_API_GUIDE.md) — Stack management API
- [GitOps Guide](gitops.md) — CI/CD pipeline details