homelab-optimized/docs/WATCHTOWER_DEPLOYMENT_FIXES.md

# Watchtower Deployment Fixes - February 2026

## Overview

This document details the comprehensive fixes applied to Watchtower auto-update configurations across all homelab hosts to resolve deployment issues and enable proper scheduled container updates.

## Problem Summary

The Authentik SSO stack deployment was failing due to Watchtower configuration issues across multiple hosts:

1. **Homelab VM**: Port conflicts and invalid notification URLs
2. **Calypso**: Configuration conflicts between polling and scheduled modes
3. **Atlantis**: Container dependency conflicts causing restart loops

## Solutions Implemented

### 1. Homelab VM Fixes (Commit: a863a9c4)

**Issues Resolved:**
- Port conflict on 8080 (conflicted with other services)
- Invalid notification URLs causing startup failures
- Missing HTTP API configuration

**Changes Made:**
```yaml
# Port mapping changed from 8080 to 8083
ports:
  - "8083:8080"

# Fixed notification URLs
WATCHTOWER_NOTIFICATIONS: gotify
WATCHTOWER_NOTIFICATION_GOTIFY_URL: "http://gotify.homelab.local/message"
WATCHTOWER_NOTIFICATION_GOTIFY_TOKEN: REDACTED_TOKEN

# Added HTTP API configuration
WATCHTOWER_HTTP_API_METRICS: true
WATCHTOWER_HTTP_API_TOKEN: "REDACTED_HTTP_TOKEN"
```

**Result:** ✅ Scheduled runs enabled at 04:00 PST daily

### 2. Calypso Fixes

**Issues Resolved:**
- Configuration conflicts between `WATCHTOWER_POLL_INTERVAL` and scheduled runs
- HTTP API update conflicts with periodic scheduling

**Changes Made:**
```yaml
# Removed conflicting settings
# WATCHTOWER_POLL_INTERVAL: 300 (removed)
# WATCHTOWER_HTTP_API_UPDATE: false (removed)

# Maintained schedule configuration
WATCHTOWER_SCHEDULE: "0 4 * * *"  # 04:00 PST daily
```

**Result:** ✅ Scheduled runs enabled at 04:00 PST daily

### 3. Atlantis Fixes (Commit: c8f4d87b)

**Issues Resolved:**
- Container dependency conflicts with deluge container
- Missing port mapping for HTTP API access
- Environment variable token resolution issues
- Network connectivity problems

**Changes Made:**
```yaml
# Disabled rolling restart to fix dependency conflicts
WATCHTOWER_ROLLING_RESTART: false

# Added port mapping for HTTP API
ports:
  - "8082:8080"

# Hardcoded token instead of environment variable
WATCHTOWER_HTTP_API_TOKEN: "REDACTED_HTTP_TOKEN"

# Created prometheus-net network
networks:
  - prometheus-net
```

**Network Setup:**
```bash
# Created Docker network on Atlantis
sudo docker network create prometheus-net
```

**Result:** ✅ Scheduled runs enabled at 02:00 PST daily

## Current Deployment Status

| Host | Status | Schedule | Port | Network | Token |
|------|--------|----------|------|---------|-------|
| **Homelab VM** | ✅ Running | 04:00 PST | 8083 | bridge | REDACTED_WATCHTOWER_TOKEN |
| **Calypso** | ✅ Running | 04:00 PST | 8080 | bridge | REDACTED_WATCHTOWER_TOKEN |
| **Atlantis** | ✅ Running | 02:00 PST | 8082 | prometheus-net | REDACTED_WATCHTOWER_TOKEN |

## Configuration Best Practices Established

### 1. Scheduling Strategy
- **Staggered schedules** to prevent simultaneous updates across hosts
- **Atlantis**: 02:00 PST (lowest priority services)
- **Homelab VM & Calypso**: 04:00 PST (critical services)

### 2. Port Management
- **Unique ports** per host to prevent conflicts
- **Consistent API access** across all deployments
- **Documented port assignments** in configuration files

### 3. Dependency Management
- **Disabled rolling restart** where container dependencies exist
- **Network isolation** using dedicated Docker networks
- **Graceful shutdown timeouts** (30 seconds) for clean restarts

### 4. Authentication & Security
- **Consistent token usage** across all deployments
- **HTTP API metrics** enabled for monitoring integration
- **Secure network configurations** with proper isolation

## Monitoring & Verification

### HTTP API Endpoints
```bash
# Homelab VM
curl -H "Authorization: Bearer REDACTED_WATCHTOWER_TOKEN" http://homelab-vm.local:8083/v1/update

# Calypso
curl -H "Authorization: Bearer REDACTED_WATCHTOWER_TOKEN" http://calypso.local:8080/v1/update

# Atlantis
curl -H "Authorization: Bearer REDACTED_WATCHTOWER_TOKEN" http://atlantis.local:8082/v1/update
```

### Container Status Verification
```bash
# Check running containers
docker ps | grep watchtower

# Check logs for scheduling confirmation
docker logs watchtower --tail 10
```

## Troubleshooting Guide

### Common Issues & Solutions

1. **Container Restart Loops**
   - **Cause**: Rolling restart conflicts with dependent containers
   - **Solution**: Set `WATCHTOWER_ROLLING_RESTART: false`

2. **Port Conflicts**
   - **Cause**: Multiple services using same port
   - **Solution**: Use unique port mappings per host

3. **Schedule Not Working**
   - **Cause**: Conflicting polling and schedule configurations
   - **Solution**: Remove `WATCHTOWER_POLL_INTERVAL` when using schedules

4. **Network Connectivity Issues**
   - **Cause**: Containers on different networks
   - **Solution**: Create dedicated networks or use bridge network

## Future Maintenance

### Regular Tasks
1. **Monitor logs** for successful update runs
2. **Verify HTTP API** accessibility monthly
3. **Check container health** after scheduled updates
4. **Update documentation** when configurations change

### Upgrade Considerations
- **Test configuration changes** in non-production first
- **Backup configurations** before major updates
- **Coordinate schedules** to minimize service disruption
- **Monitor resource usage** during update windows

## Related Documentation

- [Docker Compose Configuration Guide](../DOCKER_COMPOSE_GUIDE.md)
- [Network Configuration](NETWORK_SETUP.md)
- [Monitoring Setup](MONITORING_GUIDE.md)
- [Backup Procedures](BACKUP_PROCEDURES.md)

---

**Last Updated:** February 13, 2026  
**Author:** OpenHands Agent  
**Status:** Production Ready ✅