Sanitized mirror from private repository - 2026-03-26 10:25:55 UTC

docs/WATCHTOWER_DEPLOYMENT_FIXES.md

@@ -0,0 +1,191 @@
+# Watchtower Deployment Fixes - February 2026
+
+## Overview
+
+This document details the comprehensive fixes applied to Watchtower auto-update configurations across all homelab hosts to resolve deployment issues and enable proper scheduled container updates.
+
+## Problem Summary
+
+The Authentik SSO stack deployment was failing due to Watchtower configuration issues across multiple hosts:
+
+1. **Homelab VM**: Port conflicts and invalid notification URLs
+2. **Calypso**: Configuration conflicts between polling and scheduled modes
+3. **Atlantis**: Container dependency conflicts causing restart loops
+
+## Solutions Implemented
+
+### 1. Homelab VM Fixes (Commit: a863a9c4)
+
+**Issues Resolved:**
+- Port conflict on 8080 (conflicted with other services)
+- Invalid notification URLs causing startup failures
+- Missing HTTP API configuration
+
+**Changes Made:**
+```yaml
+# Port mapping changed from 8080 to 8083
+ports:
+  - "8083:8080"
+
+# Fixed notification URLs
+WATCHTOWER_NOTIFICATIONS: gotify
+WATCHTOWER_NOTIFICATION_GOTIFY_URL: "http://gotify.homelab.local/message"
+WATCHTOWER_NOTIFICATION_GOTIFY_TOKEN: REDACTED_TOKEN
+
+# Added HTTP API configuration
+WATCHTOWER_HTTP_API_METRICS: true
+WATCHTOWER_HTTP_API_TOKEN: "REDACTED_HTTP_TOKEN"
+```
+
+**Result:** ✅ Scheduled runs enabled at 04:00 PST daily
+
+### 2. Calypso Fixes
+
+**Issues Resolved:**
+- Configuration conflicts between `WATCHTOWER_POLL_INTERVAL` and scheduled runs
+- HTTP API update conflicts with periodic scheduling
+
+**Changes Made:**
+```yaml
+# Removed conflicting settings
+# WATCHTOWER_POLL_INTERVAL: 300 (removed)
+# WATCHTOWER_HTTP_API_UPDATE: false (removed)
+
+# Maintained schedule configuration
+WATCHTOWER_SCHEDULE: "0 4 * * *"  # 04:00 PST daily
+```
+
+**Result:** ✅ Scheduled runs enabled at 04:00 PST daily
+
+### 3. Atlantis Fixes (Commit: c8f4d87b)
+
+**Issues Resolved:**
+- Container dependency conflicts with deluge container
+- Missing port mapping for HTTP API access
+- Environment variable token resolution issues
+- Network connectivity problems
+
+**Changes Made:**
+```yaml
+# Disabled rolling restart to fix dependency conflicts
+WATCHTOWER_ROLLING_RESTART: false
+
+# Added port mapping for HTTP API
+ports:
+  - "8082:8080"
+
+# Hardcoded token instead of environment variable
+WATCHTOWER_HTTP_API_TOKEN: "REDACTED_HTTP_TOKEN"
+
+# Created prometheus-net network
+networks:
+  - prometheus-net
+```
+
+**Network Setup:**
+```bash
+# Created Docker network on Atlantis
+sudo docker network create prometheus-net
+```
+
+**Result:** ✅ Scheduled runs enabled at 02:00 PST daily
+
+## Current Deployment Status
+
+| Host | Status | Schedule | Port | Network | Token |
+|------|--------|----------|------|---------|-------|
+| **Homelab VM** | ✅ Running | 04:00 PST | 8083 | bridge | REDACTED_WATCHTOWER_TOKEN |
+| **Calypso** | ✅ Running | 04:00 PST | 8080 | bridge | REDACTED_WATCHTOWER_TOKEN |
+| **Atlantis** | ✅ Running | 02:00 PST | 8082 | prometheus-net | REDACTED_WATCHTOWER_TOKEN |
+
+## Configuration Best Practices Established
+
+### 1. Scheduling Strategy
+- **Staggered schedules** to prevent simultaneous updates across hosts
+- **Atlantis**: 02:00 PST (lowest priority services)
+- **Homelab VM & Calypso**: 04:00 PST (critical services)
+
+### 2. Port Management
+- **Unique ports** per host to prevent conflicts
+- **Consistent API access** across all deployments
+- **Documented port assignments** in configuration files
+
+### 3. Dependency Management
+- **Disabled rolling restart** where container dependencies exist
+- **Network isolation** using dedicated Docker networks
+- **Graceful shutdown timeouts** (30 seconds) for clean restarts
+
+### 4. Authentication & Security
+- **Consistent token usage** across all deployments
+- **HTTP API metrics** enabled for monitoring integration
+- **Secure network configurations** with proper isolation
+
+## Monitoring & Verification
+
+### HTTP API Endpoints
+```bash
+# Homelab VM
+curl -H "Authorization: Bearer REDACTED_WATCHTOWER_TOKEN" http://homelab-vm.local:8083/v1/update
+
+# Calypso
+curl -H "Authorization: Bearer REDACTED_WATCHTOWER_TOKEN" http://calypso.local:8080/v1/update
+
+# Atlantis
+curl -H "Authorization: Bearer REDACTED_WATCHTOWER_TOKEN" http://atlantis.local:8082/v1/update
+```
+
+### Container Status Verification
+```bash
+# Check running containers
+docker ps | grep watchtower
+
+# Check logs for scheduling confirmation
+docker logs watchtower --tail 10
+```
+
+## Troubleshooting Guide
+
+### Common Issues & Solutions
+
+1. **Container Restart Loops**
+   - **Cause**: Rolling restart conflicts with dependent containers
+   - **Solution**: Set `WATCHTOWER_ROLLING_RESTART: false`
+
+2. **Port Conflicts**
+   - **Cause**: Multiple services using same port
+   - **Solution**: Use unique port mappings per host
+
+3. **Schedule Not Working**
+   - **Cause**: Conflicting polling and schedule configurations
+   - **Solution**: Remove `WATCHTOWER_POLL_INTERVAL` when using schedules
+
+4. **Network Connectivity Issues**
+   - **Cause**: Containers on different networks
+   - **Solution**: Create dedicated networks or use bridge network
+
+## Future Maintenance
+
+### Regular Tasks
+1. **Monitor logs** for successful update runs
+2. **Verify HTTP API** accessibility monthly
+3. **Check container health** after scheduled updates
+4. **Update documentation** when configurations change
+
+### Upgrade Considerations
+- **Test configuration changes** in non-production first
+- **Backup configurations** before major updates
+- **Coordinate schedules** to minimize service disruption
+- **Monitor resource usage** during update windows
+
+## Related Documentation
+
+- [Docker Compose Configuration Guide](../DOCKER_COMPOSE_GUIDE.md)
+- [Network Configuration](NETWORK_SETUP.md)
+- [Monitoring Setup](MONITORING_GUIDE.md)
+- [Backup Procedures](BACKUP_PROCEDURES.md)
+
+---
+
+**Last Updated:** February 13, 2026  
+**Author:** OpenHands Agent  
+**Status:** Production Ready ✅