Sanitized mirror from private repository - 2026-03-20 08:50:12 UTC

docs/troubleshooting/WATCHTOWER_NOTIFICATION_FIX.md

@@ -0,0 +1,119 @@
+# Watchtower Notification Fix Guide
+
+## 🚨 **CRITICAL ERROR - CRASH LOOP**
+**If Watchtower is crash looping with "unknown service 'http'" error:**
+
+```bash
+# EMERGENCY FIX - Run this immediately:
+sudo /home/homelab/organized/repos/homelab/scripts/emergency-fix-watchtower-crash.sh
+```
+
+**Root Cause**: Using `http://` instead of `ntfy://` in WATCHTOWER_NOTIFICATION_URL causes Shoutrrr to fail with "unknown service 'http'" error.
+
+## 🚨 **Issue Identified**
+```
+error="failed to send ntfy notification: error sending payload: Post \"https://192.168.0.210:8081/updates\": http: server gave HTTP response to HTTPS client"
+```
+
+## 🔍 **Root Cause**
+- Watchtower is using `ntfy://192.168.0.210:8081/updates` 
+- The `ntfy://` protocol defaults to HTTPS
+- Your ntfy server is running on HTTP (port 8081)
+- This causes the HTTPS/HTTP protocol mismatch
+
+## ✅ **Solution**
+
+### **Option 1: Fix via Portainer (Recommended)**
+1. Open Portainer web interface
+2. Go to **Stacks** → Find the **watchtower-stack**
+3. Click **Editor** 
+4. Find the line: `WATCHTOWER_NOTIFICATION_URL=ntfy://192.168.0.210:8081/updates`
+5. Change it to: `WATCHTOWER_NOTIFICATION_URL=ntfy://localhost:8081/updates?insecure=yes`
+6. Click **Update the stack**
+
+### **Option 2: Fix via Docker Command**
+```bash
+# Stop the current container
+sudo docker stop watchtower
+sudo docker rm watchtower
+
+# Recreate with correct notification URL
+sudo docker run -d \
+  --name watchtower \
+  --restart unless-stopped \
+  -p 8091:8080 \
+  -v /var/run/docker.sock:/var/run/docker.sock \
+  -e WATCHTOWER_CLEANUP=true \
+  -e WATCHTOWER_SCHEDULE="0 0 4 * * *" \
+  -e WATCHTOWER_INCLUDE_STOPPED=false \
+  -e TZ=America/Los_Angeles \
+  -e WATCHTOWER_HTTP_API_UPDATE=true \
+  -e WATCHTOWER_HTTP_API_TOKEN="REDACTED_HTTP_TOKEN" \
+  -e WATCHTOWER_NOTIFICATIONS=shoutrrr \
+  -e WATCHTOWER_NOTIFICATION_URL="ntfy://localhost:8081/updates?insecure=yes" \
+  containrrr/watchtower:latest
+```
+
+## 🧪 **Test the Fix**
+
+### **Test ntfy Endpoints**
+```bash
+# Run comprehensive ntfy test
+./scripts/test-ntfy-notifications.sh
+
+# Or test manually:
+curl -d "Test message" http://localhost:8081/updates
+curl -d "Test message" http://192.168.0.210:8081/updates  
+curl -d "Test message" https://ntfy.vish.gg/REDACTED_NTFY_TOPIC
+```
+
+### **Test Watchtower Notifications**
+```bash
+# Trigger a manual update
+curl -H "Authorization: Bearer watchtower-update-token" \
+     -X POST http://localhost:8091/v1/update
+
+# Check logs for success (should see no HTTPS errors)
+sudo docker logs watchtower --since 30s
+```
+
+## 🎯 **Notification Options**
+
+You have **3 working ntfy endpoints**:
+
+| Endpoint | URL | Protocol | Use Case |
+|----------|-----|----------|----------|
+| **Local (localhost)** | `http://localhost:8081/updates` | HTTP | Most reliable, no network deps |
+| **Local (IP)** | `http://192.168.0.210:8081/updates` | HTTP | Local network access |
+| **External** | `https://ntfy.vish.gg/REDACTED_NTFY_TOPIC` | HTTPS | Remote notifications |
+
+### **Recommended Configurations**
+
+**Option 1: Local Only (Most Reliable)**
+```yaml
+- WATCHTOWER_NOTIFICATION_URL=ntfy://localhost:8081/updates?insecure=yes
+```
+
+**Option 2: External Only (Remote Access)**
+```yaml
+- WATCHTOWER_NOTIFICATION_URL=ntfy://ntfy.vish.gg/REDACTED_NTFY_TOPIC
+```
+
+**Option 3: Both (Redundancy)**
+```yaml
+- WATCHTOWER_NOTIFICATION_URL=ntfy://localhost:8081/updates?insecure=yes,ntfy://ntfy.vish.gg/REDACTED_NTFY_TOPIC
+```
+
+## ✅ **Expected Result**
+- No more "HTTP response to HTTPS client" errors
+- Successful notifications to ntfy server
+- Updates will be posted to: http://192.168.0.210:8081/updates
+
+## 📋 **Repository Files Updated**
+- ✅ `common/watchtower-full.yaml` - Fixed notification URL
+- ✅ `scripts/fix-watchtower-notifications.sh` - Safe fix script
+- ✅ `docs/WATCHTOWER_SECURITY_ANALYSIS.md` - Security analysis
+
+## 🔗 **Related Files**
+- [Watchtower Security Analysis](WATCHTOWER_SECURITY_ANALYSIS.md)
+- [Container Diagnosis Report](CONTAINER_DIAGNOSIS_REPORT.md)