# 🔔 ntfy Notification System Documentation

**Last Updated**: January 2025  
**System Status**: Active and Operational

This document provides a complete overview of your homelab's ntfy notification system, including configuration, sources, and modification procedures.

---

## 📋 System Overview

Your homelab uses **ntfy** (pronounced "notify") as the primary notification system. It's a simple HTTP-based pub-sub notification service that sends push notifications to mobile devices and other clients.

### Key Components

| Component | Location | Port | Purpose |
|-----------|----------|------|---------|
| **ntfy Server** | homelab-vm | 8081 | Main notification server |
| **Alertmanager** | homelab-vm | 9093 | Routes monitoring alerts |
| **ntfy-bridge** | homelab-vm | 5001 | Formats alerts for ntfy |
| **signal-bridge** | homelab-vm | 5000 | Forwards critical alerts to Signal |
| **gitea-ntfy-bridge** | homelab-vm | 8095 | Git repository notifications |

### Access URLs

- **ntfy Web Interface**: http://atlantis.vish.local:8081 (internal) or https://ntfy.vish.gg (external)
- **Alertmanager**: http://atlantis.vish.local:9093
- **Grafana**: http://atlantis.vish.local:3300

---

## 🏗️ Architecture

```
┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│   Prometheus    │────▶│   Alertmanager   │────▶│   ntfy-bridge   │───▶ ntfy Server ───▶ Mobile Apps
│  (monitoring)   │     │   (routing)      │     │   (formatting)  │     │ (8081)      │
└─────────────────┘     └────────┬─────────┘     └─────────────────┘     └─────────────┘
                                 │                                              │
                                 │ (critical alerts)                           │
                                 ▼                                              │
                        ┌─────────────────┐     ┌─────────────────┐            │
                        │  signal-bridge  │────▶│   Signal API    │            │
                        │   (critical)    │     │   (encrypted)   │            │
                        └─────────────────┘     └─────────────────┘            │
                                                                                │
┌─────────────────┐     ┌──────────────────┐                                  │
│      Gitea      │────▶│ gitea-ntfy-bridge│──────────────────────────────────┘
│ (git events)    │     │   (git format)   │
└─────────────────┘     └──────────────────┘

┌─────────────────┐                                                            │
│   Watchtower    │────────────────────────────────────────────────────────────┘
│ (container upd) │
└─────────────────┘
```

---

## 🔧 Current Configuration

### ntfy Server Configuration

**File**: `/home/homelab/docker/ntfy/config/server.yml` (on homelab-vm)

Key settings:
```yaml
base-url: "https://ntfy.vish.gg"
upstream-base-url: "https://ntfy.sh"  # Required for iOS push notifications
```

**Docker Compose**: `hosts/vms/homelab-vm/ntfy.yaml`
- **Container**: `NTFY`
- **Image**: `binwiederhier/ntfy`
- **Internal Port**: 80
- **External Port**: 8081
- **Volume**: `/home/homelab/docker/ntfy:/var/cache/ntfy`

### Notification Topic

**Primary Topic**: `homelab-alerts`

All notifications are sent to this single topic, which you can subscribe to in the ntfy mobile app.

---

## 📨 Notification Sources

### 1. Monitoring Alerts (Prometheus → Alertmanager → ntfy-bridge)

**Stack**: `alerting-stack` (Portainer ID: 500)  
**Configuration**: `hosts/vms/homelab-vm/alerting.yaml`

**Alert Routing**:
- ⚠️ **Warning alerts** → ntfy only
- 🚨 **Critical alerts** → ntfy + Signal
- ✅ **Resolved alerts** → Both channels (for critical)

**ntfy-bridge Configuration**:
```python
NTFY_URL = "http://NTFY:80"
NTFY_TOPIC = "REDACTED_NTFY_TOPIC"
```

**Alert Types Currently Configured**:
- Host down/unreachable
- High CPU/Memory/Disk usage
- Service failures
- Container resource issues

### 2. Git Repository Events (Gitea → gitea-ntfy-bridge)

**Stack**: `ntfy-stack`  
**Configuration**: `hosts/vms/homelab-vm/ntfy.yaml`

**Bridge Configuration**:
```python
NTFY_URL = "https://ntfy.vish.gg"
NTFY_TOPIC = "REDACTED_NTFY_TOPIC"
```

**Supported Events**:
- Push commits
- Pull requests (opened/closed)
- Issues (created/closed)
- Releases
- Branch creation/deletion

### 3. Container Updates (Watchtower)

**Stack**: `watchtower-stack`  
**Configuration**: `common/watchtower-full.yaml`

Watchtower sends notifications directly to ntfy when containers are updated.

---

## 🛠️ How to Modify Notifications

### Changing Notification Topics

1. **For Monitoring Alerts**:
   ```bash
   # Edit the alerting stack configuration
   vim /home/homelab/organized/scripts/homelab/hosts/vms/homelab-vm/alerting.yaml
   
   # Find line 69 and change:
   NTFY_TOPIC = os.environ.get('NTFY_TOPIC', 'your-new-topic')
   ```

2. **For Git Events**:
   ```bash
   # Edit the ntfy stack configuration
   vim /home/homelab/organized/scripts/homelab/hosts/vms/homelab-vm/ntfy.yaml
   
   # Find line 33 and change:
   - NTFY_TOPIC="REDACTED_NTFY_TOPIC"
   ```

3. **Apply Changes via Portainer**:
   - Go to http://atlantis.vish.local:10000
   - Navigate to the relevant stack
   - Click "Update the stack" (GitOps will pull changes automatically)

### Adding New Alert Rules

1. **Edit Prometheus Configuration**:
   ```bash
   # The monitoring stack doesn't currently have alert rules configured
   # You would need to add them to the prometheus_config in:
   vim /home/homelab/organized/scripts/homelab/hosts/vms/homelab-vm/monitoring.yaml
   ```

2. **Add Alert Rules Section**:
   ```yaml
   rule_files:
     - "/etc/prometheus/alert-rules.yml"
   
   alerting:
     alertmanagers:
       - static_configs:
           - targets:
             - alertmanager:9093
   ```

3. **Create Alert Rules Config**:
   ```yaml
   # Add to configs section in monitoring.yaml
   alert_rules:
     content: |
       groups:
         - name: homelab-alerts
           rules:
             - alert: HighCPUUsage
               expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
               for: 5m
               labels:
                 severity: warning
               annotations:
                 summary: "High CPU usage on {{ $labels.instance }}"
                 description: "CPU usage is above 80% for 5 minutes"
   ```

### Modifying Alert Severity and Routing

**File**: `hosts/vms/homelab-vm/alerting.yaml`

1. **Change Alert Routing**:
   ```yaml
   # Lines 30-37: Modify routing rules
   routes:
     - match:
         severity: critical
       receiver: 'critical-alerts'
     - match:
         severity: warning
       receiver: 'ntfy-all'
   ```

2. **Add New Receivers**:
   ```yaml
   # Lines 39-50: Add new notification channels
   receivers:
     - name: 'email-alerts'
       email_configs:
         - to: 'admin@yourdomain.com'
           subject: 'Homelab Alert: {{ .GroupLabels.alertname }}'
   ```

### Customizing Notification Format

**File**: `hosts/vms/homelab-vm/alerting.yaml` (lines 85-109)

The `format_alert()` function controls how notifications appear:

```python
def format_alert(alert):
    # Customize title format
    title = f"{alertname} [{status_text}] - {instance}"
    
    # Customize message body
    body_parts = []
    if summary:
        body_parts.append(f"📊 {summary}")
    if description:
        body_parts.append(f"📝 {description}")
    
    # Add custom fields
    body_parts.append(f"🕐 {datetime.now().strftime('%H:%M:%S')}")
    
    return title, body, severity, status
```

---

## 📱 Mobile App Setup

### iOS Setup

1. **Install ntfy app** from the App Store
2. **Add subscription**:
   - Server: `https://ntfy.vish.gg`
   - Topic: `homelab-alerts`
3. **Enable notifications** in iOS Settings
4. **Important**: The server must have `upstream-base-url: "https://ntfy.sh"` configured for iOS push notifications to work

### Android Setup

1. **Install ntfy app** from Google Play Store or F-Droid
2. **Add subscription**:
   - Server: `https://ntfy.vish.gg`
   - Topic: `homelab-alerts`
3. **Configure notification settings** as desired

### Web Interface

Access the web interface at:
- Internal: http://atlantis.vish.local:8081
- External: https://ntfy.vish.gg

---

## 🧪 Testing Notifications

### Test Scripts Available

**Location**: `/home/homelab/organized/scripts/homelab/scripts/test-ntfy-notifications.sh`

### Manual Testing

1. **Test Direct ntfy**:
   ```bash
   curl -H "Title: Test Alert" -d "This is a test notification" https://ntfy.vish.gg/REDACTED_NTFY_TOPIC
   ```

2. **Test Alert Bridge**:
   ```bash
   curl -X POST http://atlantis.vish.local:5001/alert -H "Content-Type: application/json" -d '{
     "alerts": [{
       "status": "firing",
       "labels": {"alertname": "TestAlert", "severity": "warning", "instance": "test:9100"},
       "annotations": {"summary": "Test alert", "description": "This is a test notification"}
     }]
   }'
   ```

3. **Test Signal Bridge** (for critical alerts):
   ```bash
   curl -X POST http://atlantis.vish.local:5000/alert -H "Content-Type: application/json" -d '{
     "alerts": [{
       "status": "firing",
       "labels": {"alertname": "TestAlert", "severity": "critical", "instance": "test:9100"},
       "annotations": {"summary": "Critical test alert", "description": "This is a critical test"}
     }]
   }'
   ```

4. **Test Gitea Bridge**:
   ```bash
   curl -X POST http://atlantis.vish.local:8095 -H "X-Gitea-Event: push" -H "Content-Type: application/json" -d '{
     "repository": {"full_name": "test/repo"},
     "sender": {"login": "testuser"},
     "commits": [{"message": "Test commit"}],
     "ref": "refs/heads/main"
   }'
   ```

---

## 🔍 Troubleshooting

### Common Issues

1. **Notifications not received on iOS**:
   - Verify `upstream-base-url: "https://ntfy.sh"` is set in server config
   - Restart ntfy container: `docker restart NTFY`
   - Re-subscribe in iOS app

2. **Alerts not firing**:
   - Check Prometheus targets: http://atlantis.vish.local:9090/targets
   - Check Alertmanager: http://atlantis.vish.local:9093
   - Verify bridge health: `curl http://atlantis.vish.local:5001/health`

3. **Signal notifications not working**:
   - Check signal-api container: `docker logs signal-api`
   - Test signal-bridge: `curl http://atlantis.vish.local:5000/health`

### Container Status Check

```bash
# Via Portainer API
curl -s -H "X-API-Key: "REDACTED_API_KEY" \
  "http://atlantis.vish.local:10000/api/endpoints/443399/docker/containers/json" | \
  jq '.[] | select(.Names[0] | contains("ntfy") or contains("alert")) | {Names: .Names, State: .State, Status: .Status}'
```

### Log Access

- **ntfy logs**: Check via Portainer → Containers → NTFY → Logs
- **Bridge logs**: Check via Portainer → Containers → ntfy-bridge → Logs
- **Alertmanager logs**: Check via Portainer → Containers → alertmanager → Logs

---

## 📊 Current Deployment Status

### Portainer Stacks

| Stack Name | Status | Endpoint | Configuration File |
|------------|--------|----------|-------------------|
| **ntfy-stack** | ✅ Running | homelab-vm (443399) | `hosts/vms/homelab-vm/ntfy.yaml` |
| **alerting-stack** | ✅ Running | homelab-vm (443399) | `hosts/vms/homelab-vm/alerting.yaml` |
| **monitoring-stack** | ✅ Running | homelab-vm (443399) | `hosts/vms/homelab-vm/monitoring.yaml` |
| **signal-api-stack** | ✅ Running | homelab-vm (443399) | `hosts/vms/homelab-vm/signal_api.yaml` |

### Container Health

| Container | Image | Status | Purpose |
|-----------|-------|--------|---------|
| **NTFY** | binwiederhier/ntfy | ✅ Running | Main notification server |
| **alertmanager** | prom/alertmanager:latest | ✅ Running | Alert routing |
| **ntfy-bridge** | python:3.11-slim | ✅ Running (healthy) | Alert formatting |
| **signal-bridge** | python:3.11-slim | ✅ Running (healthy) | Signal forwarding |
| **gitea-ntfy-bridge** | python:3.12-alpine | ✅ Running | Git notifications |
| **prometheus** | prom/prometheus:latest | ✅ Running | Metrics collection |
| **grafana** | grafana/grafana-oss:latest | ✅ Running | Monitoring dashboard |

---

## 🔐 Security Considerations

1. **ntfy Server**: Publicly accessible at https://ntfy.vish.gg
2. **Topic Security**: Uses a single topic `homelab-alerts` - consider authentication if needed
3. **Signal Integration**: Uses encrypted Signal messaging for critical alerts
4. **Internal Network**: Most bridges communicate over internal Docker networks

---

## 📚 Additional Resources

- **ntfy Documentation**: https://ntfy.sh/REDACTED_TOPIC/
- **Alertmanager Documentation**: https://prometheus.io/docs/alerting/latest/alertmanager/
- **Prometheus Alerting**: https://prometheus.io/docs/alerting/latest/rules/

---

## 🔄 Maintenance Tasks

### Regular Maintenance

1. **Monthly**: Check container health and logs
2. **Quarterly**: Test all notification channels
3. **As needed**: Update notification rules based on infrastructure changes

### Backup Important Configs

```bash
# Backup ntfy configuration
cp /home/homelab/docker/ntfy/config/server.yml /backup/ntfy-config-$(date +%Y%m%d).yml

# Backup alerting configuration (already in Git)
git -C /home/homelab/organized/scripts/homelab status
```

---

*This documentation reflects the current state of your ntfy notification system as of January 2025. For the most up-to-date configuration, always refer to the actual configuration files in the homelab Git repository.*