# Zot — OCI Pull-Through Registry Cache

## Overview

Zot is a single-container OCI registry running on Atlantis that acts as a
pull-through cache for Docker Hub (`docker.io`).

**Why:** Docker Hub rate-limits unauthenticated pulls (100/6h per IP). After the
first pull, any Docker Hub image is served instantly from local cache — no upstream
latency on deploys, watchtower updates, or container restarts.

| | |
|---|---|
| **Host** | Atlantis |
| **Port** | 5050 (5000 was taken by nginx) |
| **Web UI** | http://100.83.230.112:5050 |
| **Metrics** | http://100.83.230.112:5050/metrics |
| **Compose** | `hosts/synology/atlantis/zot.yaml` |
| **Config** | `hosts/synology/atlantis/zot/config.json` |
| **Data** | `/volume2/metadata/docker2/zot/data/` on Atlantis |

---

## Scope — What Zot Caches

**Zot caches Docker Hub images only.**

Docker's `registry-mirrors` mechanism — the standard way to redirect pulls through
a local cache — only intercepts pulls from the **default registry (Docker Hub)**.
When a compose file explicitly names a registry (e.g. `lscr.io/linuxserver/sonarr`,
`ghcr.io/immich-app/immich-server`), Docker contacts that registry directly and
bypasses the mirror entirely.

| Image type | Example | Goes through Zot? |
|---|---|---|
| Unqualified Docker Hub | `postgres:16`, `nginx:alpine`, `redis:7` | ✅ Yes |
| Explicit Docker Hub | `docker.io/library/postgres:16` | ✅ Yes |
| LinuxServer | `lscr.io/linuxserver/sonarr:latest` | ❌ No — direct |
| GitHub packages | `ghcr.io/immich-app/immich-server:release` | ❌ No — direct |
| Quay | `quay.io/prometheus/node-exporter:latest` | ❌ No — direct |

**What this means in practice:** Official images (`postgres`, `redis`, `nginx`,
`alpine`, `mariadb`, `mosquitto`, etc.) are cached. LinuxServer, Immich, Authentik,
Tdarr, and all other explicitly-prefixed images are not.

### Expanding scope (future option)

To cache `lscr.io`/`ghcr.io`/`quay.io` images, all compose files referencing those
images would need to be rewritten to pull from Zot directly (e.g.
`100.83.230.112:5050/linuxserver/sonarr:latest`), and Zot's sync config updated to
poll those registries. This is ~60 compose file changes across all hosts — deferred
for now.

---

## How It Works

```
docker pull postgres:16
       │
       ▼
Docker daemon checks registry-mirrors first
       │
       ▼
Zot (100.83.230.112:5050)
  ├── cached? → serve instantly from local disk
  └── not cached? → fetch from registry-1.docker.io, cache, serve

docker pull lscr.io/linuxserver/sonarr:latest
       │
       ▼
Docker sees explicit registry prefix → bypasses mirror
       │
       ▼
lscr.io (direct, not cached)
```

All registries are configured with `onDemand: true` — nothing is pre-downloaded.
Images are only cached when first requested, then served locally forever after
(until GC removes unreferenced blobs after 24h).

**Docker Hub note:** Docker Hub does not support catalog listing, so poll mode
cannot be used with it. On-demand only is correct.

---

## Storage

Images are stored deduplicated at `/volume2/metadata/docker2/zot/data/`.
GC runs every 24h to remove blobs no longer referenced by any cached manifest.

---

## Per-Host Mirror Configuration

Each Docker host has been configured with Zot as a registry mirror.
This is a Docker daemon setting — done once per host, not managed by Portainer.

### Status

| Host | Configured | Method |
|---|---|---|
| Atlantis | ✅ Done (manual) | DSM Container Manager → Registry → Settings → Mirror → `http://localhost:5050` |
| Calypso | ✅ Done (manual) | DSM Container Manager → Registry → Settings → Mirror → `http://100.83.230.112:5050` |
| homelab-vm | ✅ Done | `/etc/docker/daemon.json` |
| NUC | ✅ Done | `/etc/docker/daemon.json` |
| Pi-5 | ✅ Done | `/etc/docker/daemon.json` |

### daemon.json format (Linux hosts)

```json
{
  "registry-mirrors": ["http://100.83.230.112:5050"],
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" }
}
```

After editing, restart Docker: `sudo systemctl restart docker`

---

## Adding Credentials (optional)

Without credentials, public Docker Hub images pull fine but rate-limits apply
(100 pulls/6h per IP). With a Docker Hub account: 200/hr per account.

To add credentials, create this file **directly on Atlantis** (never in git):

```bash
cat > /volume2/metadata/docker2/zot/credentials.json << 'EOF'
{
  "registry-1.docker.io": {
    "username": "your-dockerhub-username",
    "password": "your-dockerhub-token"  // pragma: allowlist secret
  }
}
EOF
```

Then uncomment the credentials volume mount in `zot.yaml` and add
`"credentialsFile": "/etc/zot/credentials.json"` to the `sync` block in
`config.json`. Restart the stack via Portainer.

---

## Verifying It Works

```bash
# Zot health check
curl http://100.83.230.112:5050/v2/
# Returns empty body with HTTP 200

# View cached images
curl http://100.83.230.112:5050/v2/_catalog

# Pull a Docker Hub image on any configured host, then check catalog
docker pull alpine:latest
curl http://100.83.230.112:5050/v2/_catalog
# "library/alpine" should appear
```

---

## Known Limitations

- **lscr.io / ghcr.io / quay.io images bypass the cache** — Docker's mirror
  mechanism only intercepts unqualified (Docker Hub) pulls. See Scope section above.
- **Port 5050** — port 5000 was already in use by nginx on Atlantis.
- **No TLS** — Zot is internal-only (LAN + Tailscale). Not exposed via NPM.
- **No authentication** — read-only, internal network access only.

---

## Prometheus Integration

Zot exposes metrics at `/metrics`. Add to Prometheus scrape config:

```yaml
- job_name: 'zot'
  static_configs:
    - targets: ['100.83.230.112:5050']
  metrics_path: '/metrics'
```