119 lines
3.9 KiB
Markdown
119 lines
3.9 KiB
Markdown
# Watchtower Notification Fix Guide
|
|
|
|
## 🚨 **CRITICAL ERROR - CRASH LOOP**
|
|
**If Watchtower is crash looping with "unknown service 'http'" error:**
|
|
|
|
```bash
|
|
# EMERGENCY FIX - Run this immediately:
|
|
sudo /home/homelab/organized/repos/homelab/scripts/emergency-fix-watchtower-crash.sh
|
|
```
|
|
|
|
**Root Cause**: Using `http://` instead of `ntfy://` in WATCHTOWER_NOTIFICATION_URL causes Shoutrrr to fail with "unknown service 'http'" error.
|
|
|
|
## 🚨 **Issue Identified**
|
|
```
|
|
error="failed to send ntfy notification: error sending payload: Post \"https://192.168.0.210:8081/updates\": http: server gave HTTP response to HTTPS client"
|
|
```
|
|
|
|
## 🔍 **Root Cause**
|
|
- Watchtower is using `ntfy://192.168.0.210:8081/updates`
|
|
- The `ntfy://` protocol defaults to HTTPS
|
|
- Your ntfy server is running on HTTP (port 8081)
|
|
- This causes the HTTPS/HTTP protocol mismatch
|
|
|
|
## ✅ **Solution**
|
|
|
|
### **Option 1: Fix via Portainer (Recommended)**
|
|
1. Open Portainer web interface
|
|
2. Go to **Stacks** → Find the **watchtower-stack**
|
|
3. Click **Editor**
|
|
4. Find the line: `WATCHTOWER_NOTIFICATION_URL=ntfy://192.168.0.210:8081/updates`
|
|
5. Change it to: `WATCHTOWER_NOTIFICATION_URL=ntfy://localhost:8081/updates?insecure=yes`
|
|
6. Click **Update the stack**
|
|
|
|
### **Option 2: Fix via Docker Command**
|
|
```bash
|
|
# Stop the current container
|
|
sudo docker stop watchtower
|
|
sudo docker rm watchtower
|
|
|
|
# Recreate with correct notification URL
|
|
sudo docker run -d \
|
|
--name watchtower \
|
|
--restart unless-stopped \
|
|
-p 8091:8080 \
|
|
-v /var/run/docker.sock:/var/run/docker.sock \
|
|
-e WATCHTOWER_CLEANUP=true \
|
|
-e WATCHTOWER_SCHEDULE="0 0 4 * * *" \
|
|
-e WATCHTOWER_INCLUDE_STOPPED=false \
|
|
-e TZ=America/Los_Angeles \
|
|
-e WATCHTOWER_HTTP_API_UPDATE=true \
|
|
-e WATCHTOWER_HTTP_API_TOKEN="REDACTED_HTTP_TOKEN" \
|
|
-e WATCHTOWER_NOTIFICATIONS=shoutrrr \
|
|
-e WATCHTOWER_NOTIFICATION_URL="ntfy://localhost:8081/updates?insecure=yes" \
|
|
containrrr/watchtower:latest
|
|
```
|
|
|
|
## 🧪 **Test the Fix**
|
|
|
|
### **Test ntfy Endpoints**
|
|
```bash
|
|
# Run comprehensive ntfy test
|
|
./scripts/test-ntfy-notifications.sh
|
|
|
|
# Or test manually:
|
|
curl -d "Test message" http://localhost:8081/updates
|
|
curl -d "Test message" http://192.168.0.210:8081/updates
|
|
curl -d "Test message" https://ntfy.vish.gg/REDACTED_NTFY_TOPIC
|
|
```
|
|
|
|
### **Test Watchtower Notifications**
|
|
```bash
|
|
# Trigger a manual update
|
|
curl -H "Authorization: Bearer watchtower-update-token" \
|
|
-X POST http://localhost:8091/v1/update
|
|
|
|
# Check logs for success (should see no HTTPS errors)
|
|
sudo docker logs watchtower --since 30s
|
|
```
|
|
|
|
## 🎯 **Notification Options**
|
|
|
|
You have **3 working ntfy endpoints**:
|
|
|
|
| Endpoint | URL | Protocol | Use Case |
|
|
|----------|-----|----------|----------|
|
|
| **Local (localhost)** | `http://localhost:8081/updates` | HTTP | Most reliable, no network deps |
|
|
| **Local (IP)** | `http://192.168.0.210:8081/updates` | HTTP | Local network access |
|
|
| **External** | `https://ntfy.vish.gg/REDACTED_NTFY_TOPIC` | HTTPS | Remote notifications |
|
|
|
|
### **Recommended Configurations**
|
|
|
|
**Option 1: Local Only (Most Reliable)**
|
|
```yaml
|
|
- WATCHTOWER_NOTIFICATION_URL=ntfy://localhost:8081/updates?insecure=yes
|
|
```
|
|
|
|
**Option 2: External Only (Remote Access)**
|
|
```yaml
|
|
- WATCHTOWER_NOTIFICATION_URL=ntfy://ntfy.vish.gg/REDACTED_NTFY_TOPIC
|
|
```
|
|
|
|
**Option 3: Both (Redundancy)**
|
|
```yaml
|
|
- WATCHTOWER_NOTIFICATION_URL=ntfy://localhost:8081/updates?insecure=yes,ntfy://ntfy.vish.gg/REDACTED_NTFY_TOPIC
|
|
```
|
|
|
|
## ✅ **Expected Result**
|
|
- No more "HTTP response to HTTPS client" errors
|
|
- Successful notifications to ntfy server
|
|
- Updates will be posted to: http://192.168.0.210:8081/updates
|
|
|
|
## 📋 **Repository Files Updated**
|
|
- ✅ `common/watchtower-full.yaml` - Fixed notification URL
|
|
- ✅ `scripts/fix-watchtower-notifications.sh` - Safe fix script
|
|
- ✅ `docs/WATCHTOWER_SECURITY_ANALYSIS.md` - Security analysis
|
|
|
|
## 🔗 **Related Files**
|
|
- [Watchtower Security Analysis](WATCHTOWER_SECURITY_ANALYSIS.md)
|
|
- [Container Diagnosis Report](CONTAINER_DIAGNOSIS_REPORT.md) |