homelab-optimized/docs/troubleshooting/WATCHTOWER_NOTIFICATION_FIX.md

Watchtower Notification Fix Guide

🚨 CRITICAL ERROR - CRASH LOOP

If Watchtower is crash looping with "unknown service 'http'" error:

# EMERGENCY FIX - Run this immediately:
sudo /home/homelab/organized/repos/homelab/scripts/emergency-fix-watchtower-crash.sh

Root Cause: Using http:// instead of ntfy:// in WATCHTOWER_NOTIFICATION_URL causes Shoutrrr to fail with "unknown service 'http'" error.

🚨 Issue Identified

error="failed to send ntfy notification: error sending payload: Post \"https://192.168.0.210:8081/updates\": http: server gave HTTP response to HTTPS client"

🔍 Root Cause

• Watchtower is using ntfy://192.168.0.210:8081/updates
• The ntfy:// protocol defaults to HTTPS
• Your ntfy server is running on HTTP (port 8081)
• This causes the HTTPS/HTTP protocol mismatch

✅ Solution

Option 1: Fix via Portainer (Recommended)

1. Open Portainer web interface
2. Go to Stacks → Find the watchtower-stack
3. Click Editor
4. Find the line: WATCHTOWER_NOTIFICATION_URL=ntfy://192.168.0.210:8081/updates
5. Change it to: WATCHTOWER_NOTIFICATION_URL=ntfy://localhost:8081/updates?insecure=yes
6. Click Update the stack

Option 2: Fix via Docker Command

# Stop the current container
sudo docker stop watchtower
sudo docker rm watchtower

# Recreate with correct notification URL
sudo docker run -d \
  --name watchtower \
  --restart unless-stopped \
  -p 8091:8080 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -e WATCHTOWER_CLEANUP=true \
  -e WATCHTOWER_SCHEDULE="0 0 4 * * *" \
  -e WATCHTOWER_INCLUDE_STOPPED=false \
  -e TZ=America/Los_Angeles \
  -e WATCHTOWER_HTTP_API_UPDATE=true \
  -e WATCHTOWER_HTTP_API_TOKEN="REDACTED_HTTP_TOKEN" \
  -e WATCHTOWER_NOTIFICATIONS=shoutrrr \
  -e WATCHTOWER_NOTIFICATION_URL="ntfy://localhost:8081/updates?insecure=yes" \
  containrrr/watchtower:latest

🧪 Test the Fix

Test ntfy Endpoints

# Run comprehensive ntfy test
./scripts/test-ntfy-notifications.sh

# Or test manually:
curl -d "Test message" http://localhost:8081/updates
curl -d "Test message" http://192.168.0.210:8081/updates  
curl -d "Test message" https://ntfy.vish.gg/REDACTED_NTFY_TOPIC

Test Watchtower Notifications

# Trigger a manual update
curl -H "Authorization: Bearer watchtower-update-token" \
     -X POST http://localhost:8091/v1/update

# Check logs for success (should see no HTTPS errors)
sudo docker logs watchtower --since 30s

🎯 Notification Options

You have 3 working ntfy endpoints:

Endpoint	URL	Protocol	Use Case
Local (localhost)	http://localhost:8081/updates	HTTP	Most reliable, no network deps
Local (IP)	http://192.168.0.210:8081/updates	HTTP	Local network access
External	https://ntfy.vish.gg/REDACTED_NTFY_TOPIC	HTTPS	Remote notifications

Recommended Configurations

Option 1: Local Only (Most Reliable)

- WATCHTOWER_NOTIFICATION_URL=ntfy://localhost:8081/updates?insecure=yes

Option 2: External Only (Remote Access)

- WATCHTOWER_NOTIFICATION_URL=ntfy://ntfy.vish.gg/REDACTED_NTFY_TOPIC

Option 3: Both (Redundancy)

- WATCHTOWER_NOTIFICATION_URL=ntfy://localhost:8081/updates?insecure=yes,ntfy://ntfy.vish.gg/REDACTED_NTFY_TOPIC

✅ Expected Result

• No more "HTTP response to HTTPS client" errors
• Successful notifications to ntfy server
• Updates will be posted to: http://192.168.0.210:8081/updates

📋 Repository Files Updated

• ✅ common/watchtower-full.yaml - Fixed notification URL
• ✅ scripts/fix-watchtower-notifications.sh - Safe fix script
• ✅ docs/WATCHTOWER_SECURITY_ANALYSIS.md - Security analysis

🔗 Related Files

• Watchtower Security Analysis
• Container Diagnosis Report