homelab-optimized/docs/WATCHTOWER_DEPLOYMENT_FIXES.md

Watchtower Deployment Fixes - February 2026

Overview

This document details the comprehensive fixes applied to Watchtower auto-update configurations across all homelab hosts to resolve deployment issues and enable proper scheduled container updates.

Problem Summary

The Authentik SSO stack deployment was failing due to Watchtower configuration issues across multiple hosts:

1. Homelab VM: Port conflicts and invalid notification URLs
2. Calypso: Configuration conflicts between polling and scheduled modes
3. Atlantis: Container dependency conflicts causing restart loops

Solutions Implemented

1. Homelab VM Fixes (Commit: a863a9c4)

Issues Resolved:

• Port conflict on 8080 (conflicted with other services)
• Invalid notification URLs causing startup failures
• Missing HTTP API configuration

Changes Made:

# Port mapping changed from 8080 to 8083
ports:
  - "8083:8080"

# Fixed notification URLs
WATCHTOWER_NOTIFICATIONS: gotify
WATCHTOWER_NOTIFICATION_GOTIFY_URL: "http://gotify.homelab.local/message"
WATCHTOWER_NOTIFICATION_GOTIFY_TOKEN: REDACTED_TOKEN

# Added HTTP API configuration
WATCHTOWER_HTTP_API_METRICS: true
WATCHTOWER_HTTP_API_TOKEN: "REDACTED_HTTP_TOKEN"

Result: ✅ Scheduled runs enabled at 04:00 PST daily

2. Calypso Fixes

Issues Resolved:

• Configuration conflicts between WATCHTOWER_POLL_INTERVAL and scheduled runs
• HTTP API update conflicts with periodic scheduling

Changes Made:

# Removed conflicting settings
# WATCHTOWER_POLL_INTERVAL: 300 (removed)
# WATCHTOWER_HTTP_API_UPDATE: false (removed)

# Maintained schedule configuration
WATCHTOWER_SCHEDULE: "0 4 * * *"  # 04:00 PST daily

Result: ✅ Scheduled runs enabled at 04:00 PST daily

3. Atlantis Fixes (Commit: c8f4d87b)

Issues Resolved:

• Container dependency conflicts with deluge container
• Missing port mapping for HTTP API access
• Environment variable token resolution issues
• Network connectivity problems

Changes Made:

# Disabled rolling restart to fix dependency conflicts
WATCHTOWER_ROLLING_RESTART: false

# Added port mapping for HTTP API
ports:
  - "8082:8080"

# Hardcoded token instead of environment variable
WATCHTOWER_HTTP_API_TOKEN: "REDACTED_HTTP_TOKEN"

# Created prometheus-net network
networks:
  - prometheus-net

Network Setup:

# Created Docker network on Atlantis
sudo docker network create prometheus-net

Result: ✅ Scheduled runs enabled at 02:00 PST daily

Current Deployment Status

Host	Status	Schedule	Port	Network	Token
Homelab VM	✅ Running	04:00 PST	8083	bridge	REDACTED_WATCHTOWER_TOKEN
Calypso	✅ Running	04:00 PST	8080	bridge	REDACTED_WATCHTOWER_TOKEN
Atlantis	✅ Running	02:00 PST	8082	prometheus-net	REDACTED_WATCHTOWER_TOKEN

Configuration Best Practices Established

1. Scheduling Strategy

• Staggered schedules to prevent simultaneous updates across hosts
• Atlantis: 02:00 PST (lowest priority services)
• Homelab VM & Calypso: 04:00 PST (critical services)

2. Port Management

• Unique ports per host to prevent conflicts
• Consistent API access across all deployments
• Documented port assignments in configuration files

3. Dependency Management

• Disabled rolling restart where container dependencies exist
• Network isolation using dedicated Docker networks
• Graceful shutdown timeouts (30 seconds) for clean restarts

4. Authentication & Security

• Consistent token usage across all deployments
• HTTP API metrics enabled for monitoring integration
• Secure network configurations with proper isolation

Monitoring & Verification

HTTP API Endpoints

# Homelab VM
curl -H "Authorization: Bearer REDACTED_WATCHTOWER_TOKEN" http://homelab-vm.local:8083/v1/update

# Calypso
curl -H "Authorization: Bearer REDACTED_WATCHTOWER_TOKEN" http://calypso.local:8080/v1/update

# Atlantis
curl -H "Authorization: Bearer REDACTED_WATCHTOWER_TOKEN" http://atlantis.local:8082/v1/update

Container Status Verification

# Check running containers
docker ps | grep watchtower

# Check logs for scheduling confirmation
docker logs watchtower --tail 10

Troubleshooting Guide

Common Issues & Solutions

1. Container Restart Loops

   • Cause: Rolling restart conflicts with dependent containers
   • Solution: Set WATCHTOWER_ROLLING_RESTART: false

2. Port Conflicts

   • Cause: Multiple services using same port
   • Solution: Use unique port mappings per host

3. Schedule Not Working

   • Cause: Conflicting polling and schedule configurations
   • Solution: Remove WATCHTOWER_POLL_INTERVAL when using schedules

4. Network Connectivity Issues

   • Cause: Containers on different networks
   • Solution: Create dedicated networks or use bridge network

Future Maintenance

Regular Tasks

1. Monitor logs for successful update runs
2. Verify HTTP API accessibility monthly
3. Check container health after scheduled updates
4. Update documentation when configurations change

Upgrade Considerations

• Test configuration changes in non-production first
• Backup configurations before major updates
• Coordinate schedules to minimize service disruption
• Monitor resource usage during update windows

Related Documentation

• Docker Compose Configuration Guide
• Network Configuration
• Monitoring Setup
• Backup Procedures

---

Last Updated: February 13, 2026
Author: OpenHands Agent
Status: Production Ready ✅