# Watchtower Notification Fix Guide

## 🚨 **CRITICAL ERROR - CRASH LOOP**
**If Watchtower is crash looping with "unknown service 'http'" error:**

```bash
# EMERGENCY FIX - Run this immediately:
sudo /home/homelab/organized/repos/homelab/scripts/emergency-fix-watchtower-crash.sh
```

**Root Cause**: Using `http://` instead of `ntfy://` in WATCHTOWER_NOTIFICATION_URL causes Shoutrrr to fail with "unknown service 'http'" error.

## 🚨 **Issue Identified**
```
error="failed to send ntfy notification: error sending payload: Post \"https://192.168.0.210:8081/updates\": http: server gave HTTP response to HTTPS client"
```

## 🔍 **Root Cause**
- Watchtower is using `ntfy://192.168.0.210:8081/updates` 
- The `ntfy://` protocol defaults to HTTPS
- Your ntfy server is running on HTTP (port 8081)
- This causes the HTTPS/HTTP protocol mismatch

## ✅ **Solution**

### **Option 1: Fix via Portainer (Recommended)**
1. Open Portainer web interface
2. Go to **Stacks** → Find the **watchtower-stack**
3. Click **Editor** 
4. Find the line: `WATCHTOWER_NOTIFICATION_URL=ntfy://192.168.0.210:8081/updates`
5. Change it to: `WATCHTOWER_NOTIFICATION_URL=ntfy://localhost:8081/updates?insecure=yes`
6. Click **Update the stack**

### **Option 2: Fix via Docker Command**
```bash
# Stop the current container
sudo docker stop watchtower
sudo docker rm watchtower

# Recreate with correct notification URL
sudo docker run -d \
  --name watchtower \
  --restart unless-stopped \
  -p 8091:8080 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -e WATCHTOWER_CLEANUP=true \
  -e WATCHTOWER_SCHEDULE="0 0 4 * * *" \
  -e WATCHTOWER_INCLUDE_STOPPED=false \
  -e TZ=America/Los_Angeles \
  -e WATCHTOWER_HTTP_API_UPDATE=true \
  -e WATCHTOWER_HTTP_API_TOKEN="REDACTED_HTTP_TOKEN" \
  -e WATCHTOWER_NOTIFICATIONS=shoutrrr \
  -e WATCHTOWER_NOTIFICATION_URL="ntfy://localhost:8081/updates?insecure=yes" \
  containrrr/watchtower:latest
```

## 🧪 **Test the Fix**

### **Test ntfy Endpoints**
```bash
# Run comprehensive ntfy test
./scripts/test-ntfy-notifications.sh

# Or test manually:
curl -d "Test message" http://localhost:8081/updates
curl -d "Test message" http://192.168.0.210:8081/updates  
curl -d "Test message" https://ntfy.vish.gg/REDACTED_NTFY_TOPIC
```

### **Test Watchtower Notifications**
```bash
# Trigger a manual update
curl -H "Authorization: Bearer watchtower-update-token" \
     -X POST http://localhost:8091/v1/update

# Check logs for success (should see no HTTPS errors)
sudo docker logs watchtower --since 30s
```

## 🎯 **Notification Options**

You have **3 working ntfy endpoints**:

| Endpoint | URL | Protocol | Use Case |
|----------|-----|----------|----------|
| **Local (localhost)** | `http://localhost:8081/updates` | HTTP | Most reliable, no network deps |
| **Local (IP)** | `http://192.168.0.210:8081/updates` | HTTP | Local network access |
| **External** | `https://ntfy.vish.gg/REDACTED_NTFY_TOPIC` | HTTPS | Remote notifications |

### **Recommended Configurations**

**Option 1: Local Only (Most Reliable)**
```yaml
- WATCHTOWER_NOTIFICATION_URL=ntfy://localhost:8081/updates?insecure=yes
```

**Option 2: External Only (Remote Access)**
```yaml
- WATCHTOWER_NOTIFICATION_URL=ntfy://ntfy.vish.gg/REDACTED_NTFY_TOPIC
```

**Option 3: Both (Redundancy)**
```yaml
- WATCHTOWER_NOTIFICATION_URL=ntfy://localhost:8081/updates?insecure=yes,ntfy://ntfy.vish.gg/REDACTED_NTFY_TOPIC
```

## ✅ **Expected Result**
- No more "HTTP response to HTTPS client" errors
- Successful notifications to ntfy server
- Updates will be posted to: http://192.168.0.210:8081/updates

## 📋 **Repository Files Updated**
- ✅ `common/watchtower-full.yaml` - Fixed notification URL
- ✅ `scripts/fix-watchtower-notifications.sh` - Safe fix script
- ✅ `docs/WATCHTOWER_SECURITY_ANALYSIS.md` - Security analysis

## 🔗 **Related Files**
- [Watchtower Security Analysis](WATCHTOWER_SECURITY_ANALYSIS.md)
- [Container Diagnosis Report](CONTAINER_DIAGNOSIS_REPORT.md)