Sanitized mirror from private repository - 2026-04-01 03:48:17 UTC
This commit is contained in:
Gitea Mirror Bot
427
docs/admin/ntfy-notification-system.md
Normal file
427
docs/admin/ntfy-notification-system.md
Normal file
@@ -0,0 +1,427 @@
|
||||
# 🔔 ntfy Notification System Documentation
|
||||
|
||||
**Last Updated**: January 2025
|
||||
**System Status**: Active and Operational
|
||||
|
||||
This document provides a complete overview of your homelab's ntfy notification system, including configuration, sources, and modification procedures.
|
||||
|
||||
---
|
||||
|
||||
## 📋 System Overview
|
||||
|
||||
Your homelab uses **ntfy** (pronounced "notify") as the primary notification system. It's a simple HTTP-based pub-sub notification service that sends push notifications to mobile devices and other clients.
|
||||
|
||||
### Key Components
|
||||
|
||||
| Component | Location | Port | Purpose |
|
||||
|-----------|----------|------|---------|
|
||||
| **ntfy Server** | homelab-vm | 8081 | Main notification server |
|
||||
| **Alertmanager** | homelab-vm | 9093 | Routes monitoring alerts |
|
||||
| **ntfy-bridge** | homelab-vm | 5001 | Formats alerts for ntfy |
|
||||
| **signal-bridge** | homelab-vm | 5000 | Forwards critical alerts to Signal |
|
||||
| **gitea-ntfy-bridge** | homelab-vm | 8095 | Git repository notifications |
|
||||
|
||||
### Access URLs
|
||||
|
||||
- **ntfy Web Interface**: http://atlantis.vish.local:8081 (internal) or https://ntfy.vish.gg (external)
|
||||
- **Alertmanager**: http://atlantis.vish.local:9093
|
||||
- **Grafana**: http://atlantis.vish.local:3300
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
|
||||
│ Prometheus │────▶│ Alertmanager │────▶│ ntfy-bridge │───▶ ntfy Server ───▶ Mobile Apps
|
||||
│ (monitoring) │ │ (routing) │ │ (formatting) │ │ (8081) │
|
||||
└─────────────────┘ └────────┬─────────┘ └─────────────────┘ └─────────────┘
|
||||
│ │
|
||||
│ (critical alerts) │
|
||||
▼ │
|
||||
┌─────────────────┐ ┌─────────────────┐ │
|
||||
│ signal-bridge │────▶│ Signal API │ │
|
||||
│ (critical) │ │ (encrypted) │ │
|
||||
└─────────────────┘ └─────────────────┘ │
|
||||
│
|
||||
┌─────────────────┐ ┌──────────────────┐ │
|
||||
│ Gitea │────▶│ gitea-ntfy-bridge│──────────────────────────────────┘
|
||||
│ (git events) │ │ (git format) │
|
||||
└─────────────────┘ └──────────────────┘
|
||||
|
||||
┌─────────────────┐ │
|
||||
│ Watchtower │────────────────────────────────────────────────────────────┘
|
||||
│ (container upd) │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Current Configuration
|
||||
|
||||
### ntfy Server Configuration
|
||||
|
||||
**File**: `/home/homelab/docker/ntfy/config/server.yml` (on homelab-vm)
|
||||
|
||||
Key settings:
|
||||
```yaml
|
||||
base-url: "https://ntfy.vish.gg"
|
||||
upstream-base-url: "https://ntfy.sh" # Required for iOS push notifications
|
||||
```
|
||||
|
||||
**Docker Compose**: `hosts/vms/homelab-vm/ntfy.yaml`
|
||||
- **Container**: `NTFY`
|
||||
- **Image**: `binwiederhier/ntfy`
|
||||
- **Internal Port**: 80
|
||||
- **External Port**: 8081
|
||||
- **Volume**: `/home/homelab/docker/ntfy:/var/cache/ntfy`
|
||||
|
||||
### Notification Topic
|
||||
|
||||
**Primary Topic**: `homelab-alerts`
|
||||
|
||||
All notifications are sent to this single topic, which you can subscribe to in the ntfy mobile app.
|
||||
|
||||
---
|
||||
|
||||
## 📨 Notification Sources
|
||||
|
||||
### 1. Monitoring Alerts (Prometheus → Alertmanager → ntfy-bridge)
|
||||
|
||||
**Stack**: `alerting-stack` (Portainer ID: 500)
|
||||
**Configuration**: `hosts/vms/homelab-vm/alerting.yaml`
|
||||
|
||||
**Alert Routing**:
|
||||
- ⚠️ **Warning alerts** → ntfy only
|
||||
- 🚨 **Critical alerts** → ntfy + Signal
|
||||
- ✅ **Resolved alerts** → Both channels (for critical)
|
||||
|
||||
**ntfy-bridge Configuration**:
|
||||
```python
|
||||
NTFY_URL = "http://NTFY:80"
|
||||
NTFY_TOPIC = "REDACTED_NTFY_TOPIC"
|
||||
```
|
||||
|
||||
**Alert Types Currently Configured**:
|
||||
- Host down/unreachable
|
||||
- High CPU/Memory/Disk usage
|
||||
- Service failures
|
||||
- Container resource issues
|
||||
|
||||
### 2. Git Repository Events (Gitea → gitea-ntfy-bridge)
|
||||
|
||||
**Stack**: `ntfy-stack`
|
||||
**Configuration**: `hosts/vms/homelab-vm/ntfy.yaml`
|
||||
|
||||
**Bridge Configuration**:
|
||||
```python
|
||||
NTFY_URL = "https://ntfy.vish.gg"
|
||||
NTFY_TOPIC = "REDACTED_NTFY_TOPIC"
|
||||
```
|
||||
|
||||
**Supported Events**:
|
||||
- Push commits
|
||||
- Pull requests (opened/closed)
|
||||
- Issues (created/closed)
|
||||
- Releases
|
||||
- Branch creation/deletion
|
||||
|
||||
### 3. Container Updates (Watchtower)
|
||||
|
||||
**Stack**: `watchtower-stack`
|
||||
**Configuration**: `common/watchtower-full.yaml`
|
||||
|
||||
Watchtower sends notifications directly to ntfy when containers are updated.
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ How to Modify Notifications
|
||||
|
||||
### Changing Notification Topics
|
||||
|
||||
1. **For Monitoring Alerts**:
|
||||
```bash
|
||||
# Edit the alerting stack configuration
|
||||
vim /home/homelab/organized/scripts/homelab/hosts/vms/homelab-vm/alerting.yaml
|
||||
|
||||
# Find line 69 and change:
|
||||
NTFY_TOPIC = os.environ.get('NTFY_TOPIC', 'your-new-topic')
|
||||
```
|
||||
|
||||
2. **For Git Events**:
|
||||
```bash
|
||||
# Edit the ntfy stack configuration
|
||||
vim /home/homelab/organized/scripts/homelab/hosts/vms/homelab-vm/ntfy.yaml
|
||||
|
||||
# Find line 33 and change:
|
||||
- NTFY_TOPIC="REDACTED_NTFY_TOPIC"
|
||||
```
|
||||
|
||||
3. **Apply Changes via Portainer**:
|
||||
- Go to http://atlantis.vish.local:10000
|
||||
- Navigate to the relevant stack
|
||||
- Click "Update the stack" (GitOps will pull changes automatically)
|
||||
|
||||
### Adding New Alert Rules
|
||||
|
||||
1. **Edit Prometheus Configuration**:
|
||||
```bash
|
||||
# The monitoring stack doesn't currently have alert rules configured
|
||||
# You would need to add them to the prometheus_config in:
|
||||
vim /home/homelab/organized/scripts/homelab/hosts/vms/homelab-vm/monitoring.yaml
|
||||
```
|
||||
|
||||
2. **Add Alert Rules Section**:
|
||||
```yaml
|
||||
rule_files:
|
||||
- "/etc/prometheus/alert-rules.yml"
|
||||
|
||||
alerting:
|
||||
alertmanagers:
|
||||
- static_configs:
|
||||
- targets:
|
||||
- alertmanager:9093
|
||||
```
|
||||
|
||||
3. **Create Alert Rules Config**:
|
||||
```yaml
|
||||
# Add to configs section in monitoring.yaml
|
||||
alert_rules:
|
||||
content: |
|
||||
groups:
|
||||
- name: homelab-alerts
|
||||
rules:
|
||||
- alert: HighCPUUsage
|
||||
expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "High CPU usage on {{ $labels.instance }}"
|
||||
description: "CPU usage is above 80% for 5 minutes"
|
||||
```
|
||||
|
||||
### Modifying Alert Severity and Routing
|
||||
|
||||
**File**: `hosts/vms/homelab-vm/alerting.yaml`
|
||||
|
||||
1. **Change Alert Routing**:
|
||||
```yaml
|
||||
# Lines 30-37: Modify routing rules
|
||||
routes:
|
||||
- match:
|
||||
severity: critical
|
||||
receiver: 'critical-alerts'
|
||||
- match:
|
||||
severity: warning
|
||||
receiver: 'ntfy-all'
|
||||
```
|
||||
|
||||
2. **Add New Receivers**:
|
||||
```yaml
|
||||
# Lines 39-50: Add new notification channels
|
||||
receivers:
|
||||
- name: 'email-alerts'
|
||||
email_configs:
|
||||
- to: 'admin@yourdomain.com'
|
||||
subject: 'Homelab Alert: {{ .GroupLabels.alertname }}'
|
||||
```
|
||||
|
||||
### Customizing Notification Format
|
||||
|
||||
**File**: `hosts/vms/homelab-vm/alerting.yaml` (lines 85-109)
|
||||
|
||||
The `format_alert()` function controls how notifications appear:
|
||||
|
||||
```python
|
||||
def format_alert(alert):
|
||||
# Customize title format
|
||||
title = f"{alertname} [{status_text}] - {instance}"
|
||||
|
||||
# Customize message body
|
||||
body_parts = []
|
||||
if summary:
|
||||
body_parts.append(f"📊 {summary}")
|
||||
if description:
|
||||
body_parts.append(f"📝 {description}")
|
||||
|
||||
# Add custom fields
|
||||
body_parts.append(f"🕐 {datetime.now().strftime('%H:%M:%S')}")
|
||||
|
||||
return title, body, severity, status
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📱 Mobile App Setup
|
||||
|
||||
### iOS Setup
|
||||
|
||||
1. **Install ntfy app** from the App Store
|
||||
2. **Add subscription**:
|
||||
- Server: `https://ntfy.vish.gg`
|
||||
- Topic: `homelab-alerts`
|
||||
3. **Enable notifications** in iOS Settings
|
||||
4. **Important**: The server must have `upstream-base-url: "https://ntfy.sh"` configured for iOS push notifications to work
|
||||
|
||||
### Android Setup
|
||||
|
||||
1. **Install ntfy app** from Google Play Store or F-Droid
|
||||
2. **Add subscription**:
|
||||
- Server: `https://ntfy.vish.gg`
|
||||
- Topic: `homelab-alerts`
|
||||
3. **Configure notification settings** as desired
|
||||
|
||||
### Web Interface
|
||||
|
||||
Access the web interface at:
|
||||
- Internal: http://atlantis.vish.local:8081
|
||||
- External: https://ntfy.vish.gg
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Testing Notifications
|
||||
|
||||
### Test Scripts Available
|
||||
|
||||
**Location**: `/home/homelab/organized/scripts/homelab/scripts/test-ntfy-notifications.sh`
|
||||
|
||||
### Manual Testing
|
||||
|
||||
1. **Test Direct ntfy**:
|
||||
```bash
|
||||
curl -H "Title: Test Alert" -d "This is a test notification" https://ntfy.vish.gg/REDACTED_NTFY_TOPIC
|
||||
```
|
||||
|
||||
2. **Test Alert Bridge**:
|
||||
```bash
|
||||
curl -X POST http://atlantis.vish.local:5001/alert -H "Content-Type: application/json" -d '{
|
||||
"alerts": [{
|
||||
"status": "firing",
|
||||
"labels": {"alertname": "TestAlert", "severity": "warning", "instance": "test:9100"},
|
||||
"annotations": {"summary": "Test alert", "description": "This is a test notification"}
|
||||
}]
|
||||
}'
|
||||
```
|
||||
|
||||
3. **Test Signal Bridge** (for critical alerts):
|
||||
```bash
|
||||
curl -X POST http://atlantis.vish.local:5000/alert -H "Content-Type: application/json" -d '{
|
||||
"alerts": [{
|
||||
"status": "firing",
|
||||
"labels": {"alertname": "TestAlert", "severity": "critical", "instance": "test:9100"},
|
||||
"annotations": {"summary": "Critical test alert", "description": "This is a critical test"}
|
||||
}]
|
||||
}'
|
||||
```
|
||||
|
||||
4. **Test Gitea Bridge**:
|
||||
```bash
|
||||
curl -X POST http://atlantis.vish.local:8095 -H "X-Gitea-Event: push" -H "Content-Type: application/json" -d '{
|
||||
"repository": {"full_name": "test/repo"},
|
||||
"sender": {"login": "testuser"},
|
||||
"commits": [{"message": "Test commit"}],
|
||||
"ref": "refs/heads/main"
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Notifications not received on iOS**:
|
||||
- Verify `upstream-base-url: "https://ntfy.sh"` is set in server config
|
||||
- Restart ntfy container: `docker restart NTFY`
|
||||
- Re-subscribe in iOS app
|
||||
|
||||
2. **Alerts not firing**:
|
||||
- Check Prometheus targets: http://atlantis.vish.local:9090/targets
|
||||
- Check Alertmanager: http://atlantis.vish.local:9093
|
||||
- Verify bridge health: `curl http://atlantis.vish.local:5001/health`
|
||||
|
||||
3. **Signal notifications not working**:
|
||||
- Check signal-api container: `docker logs signal-api`
|
||||
- Test signal-bridge: `curl http://atlantis.vish.local:5000/health`
|
||||
|
||||
### Container Status Check
|
||||
|
||||
```bash
|
||||
# Via Portainer API
|
||||
curl -s -H "X-API-Key: "REDACTED_API_KEY" \
|
||||
"http://atlantis.vish.local:10000/api/endpoints/443399/docker/containers/json" | \
|
||||
jq '.[] | select(.Names[0] | contains("ntfy") or contains("alert")) | {Names: .Names, State: .State, Status: .Status}'
|
||||
```
|
||||
|
||||
### Log Access
|
||||
|
||||
- **ntfy logs**: Check via Portainer → Containers → NTFY → Logs
|
||||
- **Bridge logs**: Check via Portainer → Containers → ntfy-bridge → Logs
|
||||
- **Alertmanager logs**: Check via Portainer → Containers → alertmanager → Logs
|
||||
|
||||
---
|
||||
|
||||
## 📊 Current Deployment Status
|
||||
|
||||
### Portainer Stacks
|
||||
|
||||
| Stack Name | Status | Endpoint | Configuration File |
|
||||
|------------|--------|----------|-------------------|
|
||||
| **ntfy-stack** | ✅ Running | homelab-vm (443399) | `hosts/vms/homelab-vm/ntfy.yaml` |
|
||||
| **alerting-stack** | ✅ Running | homelab-vm (443399) | `hosts/vms/homelab-vm/alerting.yaml` |
|
||||
| **monitoring-stack** | ✅ Running | homelab-vm (443399) | `hosts/vms/homelab-vm/monitoring.yaml` |
|
||||
| **signal-api-stack** | ✅ Running | homelab-vm (443399) | `hosts/vms/homelab-vm/signal_api.yaml` |
|
||||
|
||||
### Container Health
|
||||
|
||||
| Container | Image | Status | Purpose |
|
||||
|-----------|-------|--------|---------|
|
||||
| **NTFY** | binwiederhier/ntfy | ✅ Running | Main notification server |
|
||||
| **alertmanager** | prom/alertmanager:latest | ✅ Running | Alert routing |
|
||||
| **ntfy-bridge** | python:3.11-slim | ✅ Running (healthy) | Alert formatting |
|
||||
| **signal-bridge** | python:3.11-slim | ✅ Running (healthy) | Signal forwarding |
|
||||
| **gitea-ntfy-bridge** | python:3.12-alpine | ✅ Running | Git notifications |
|
||||
| **prometheus** | prom/prometheus:latest | ✅ Running | Metrics collection |
|
||||
| **grafana** | grafana/grafana-oss:latest | ✅ Running | Monitoring dashboard |
|
||||
|
||||
---
|
||||
|
||||
## 🔐 Security Considerations
|
||||
|
||||
1. **ntfy Server**: Publicly accessible at https://ntfy.vish.gg
|
||||
2. **Topic Security**: Uses a single topic `homelab-alerts` - consider authentication if needed
|
||||
3. **Signal Integration**: Uses encrypted Signal messaging for critical alerts
|
||||
4. **Internal Network**: Most bridges communicate over internal Docker networks
|
||||
|
||||
---
|
||||
|
||||
## 📚 Additional Resources
|
||||
|
||||
- **ntfy Documentation**: https://ntfy.sh/REDACTED_TOPIC/
|
||||
- **Alertmanager Documentation**: https://prometheus.io/docs/alerting/latest/alertmanager/
|
||||
- **Prometheus Alerting**: https://prometheus.io/docs/alerting/latest/rules/
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Maintenance Tasks
|
||||
|
||||
### Regular Maintenance
|
||||
|
||||
1. **Monthly**: Check container health and logs
|
||||
2. **Quarterly**: Test all notification channels
|
||||
3. **As needed**: Update notification rules based on infrastructure changes
|
||||
|
||||
### Backup Important Configs
|
||||
|
||||
```bash
|
||||
# Backup ntfy configuration
|
||||
cp /home/homelab/docker/ntfy/config/server.yml /backup/ntfy-config-$(date +%Y%m%d).yml
|
||||
|
||||
# Backup alerting configuration (already in Git)
|
||||
git -C /home/homelab/organized/scripts/homelab status
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*This documentation reflects the current state of your ntfy notification system as of January 2025. For the most up-to-date configuration, always refer to the actual configuration files in the homelab Git repository.*
|
||||
Reference in New Issue
Block a user