homelab-optimized/docs/admin/ntfy-notification-system.md

🔔 ntfy Notification System Documentation

Last Updated: January 2025
System Status: Active and Operational

This document provides a complete overview of your homelab's ntfy notification system, including configuration, sources, and modification procedures.

---

📋 System Overview

Your homelab uses ntfy (pronounced "notify") as the primary notification system. It's a simple HTTP-based pub-sub notification service that sends push notifications to mobile devices and other clients.

Key Components

Component	Location	Port	Purpose
ntfy Server	homelab-vm	8081	Main notification server
Alertmanager	homelab-vm	9093	Routes monitoring alerts
ntfy-bridge	homelab-vm	5001	Formats alerts for ntfy
signal-bridge	homelab-vm	5000	Forwards critical alerts to Signal
gitea-ntfy-bridge	homelab-vm	8095	Git repository notifications

Access URLs

• ntfy Web Interface: http://atlantis.vish.local:8081 (internal) or https://ntfy.vish.gg (external)
• Alertmanager: http://atlantis.vish.local:9093
• Grafana: http://atlantis.vish.local:3300

---

🏗️ Architecture

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│   Prometheus    │────▶│   Alertmanager   │────▶│   ntfy-bridge   │───▶ ntfy Server ───▶ Mobile Apps
│  (monitoring)   │     │   (routing)      │     │   (formatting)  │     │ (8081)      │
└─────────────────┘     └────────┬─────────┘     └─────────────────┘     └─────────────┘
                                 │                                              │
                                 │ (critical alerts)                           │
                                 ▼                                              │
                        ┌─────────────────┐     ┌─────────────────┐            │
                        │  signal-bridge  │────▶│   Signal API    │            │
                        │   (critical)    │     │   (encrypted)   │            │
                        └─────────────────┘     └─────────────────┘            │
                                                                                │
┌─────────────────┐     ┌──────────────────┐                                  │
│      Gitea      │────▶│ gitea-ntfy-bridge│──────────────────────────────────┘
│ (git events)    │     │   (git format)   │
└─────────────────┘     └──────────────────┘

┌─────────────────┐                                                            │
│   Watchtower    │────────────────────────────────────────────────────────────┘
│ (container upd) │
└─────────────────┘

---

🔧 Current Configuration

ntfy Server Configuration

File: /home/homelab/docker/ntfy/config/server.yml (on homelab-vm)

Key settings:

base-url: "https://ntfy.vish.gg"
upstream-base-url: "https://ntfy.sh"  # Required for iOS push notifications

Docker Compose: hosts/vms/homelab-vm/ntfy.yaml

• Container: NTFY
• Image: binwiederhier/ntfy
• Internal Port: 80
• External Port: 8081
• Volume: /home/homelab/docker/ntfy:/var/cache/ntfy

Notification Topic

Primary Topic: homelab-alerts

All notifications are sent to this single topic, which you can subscribe to in the ntfy mobile app.

---

📨 Notification Sources

1. Monitoring Alerts (Prometheus → Alertmanager → ntfy-bridge)

Stack: alerting-stack (Portainer ID: 500)
Configuration: hosts/vms/homelab-vm/alerting.yaml

Alert Routing:

• ⚠️ Warning alerts → ntfy only
• 🚨 Critical alerts → ntfy + Signal
• ✅ Resolved alerts → Both channels (for critical)

ntfy-bridge Configuration:

NTFY_URL = "http://NTFY:80"
NTFY_TOPIC = "REDACTED_NTFY_TOPIC"

Alert Types Currently Configured:

• Host down/unreachable
• High CPU/Memory/Disk usage
• Service failures
• Container resource issues

2. Git Repository Events (Gitea → gitea-ntfy-bridge)

Stack: ntfy-stack
Configuration: hosts/vms/homelab-vm/ntfy.yaml

Bridge Configuration:

NTFY_URL = "https://ntfy.vish.gg"
NTFY_TOPIC = "REDACTED_NTFY_TOPIC"

Supported Events:

• Push commits
• Pull requests (opened/closed)
• Issues (created/closed)
• Releases
• Branch creation/deletion

3. Container Updates (Watchtower)

Stack: watchtower-stack
Configuration: common/watchtower-full.yaml

Watchtower sends notifications directly to ntfy when containers are updated.

---

🛠️ How to Modify Notifications

Changing Notification Topics

1. For Monitoring Alerts:

   # Edit the alerting stack configuration
   vim /home/homelab/organized/scripts/homelab/hosts/vms/homelab-vm/alerting.yaml
   
   # Find line 69 and change:
   NTFY_TOPIC = os.environ.get('NTFY_TOPIC', 'your-new-topic')
   

2. For Git Events:

   # Edit the ntfy stack configuration
   vim /home/homelab/organized/scripts/homelab/hosts/vms/homelab-vm/ntfy.yaml
   
   # Find line 33 and change:
   - NTFY_TOPIC="REDACTED_NTFY_TOPIC"
   

3. Apply Changes via Portainer:

   • Go to http://atlantis.vish.local:10000
   • Navigate to the relevant stack
   • Click "Update the stack" (GitOps will pull changes automatically)

Adding New Alert Rules

1. Edit Prometheus Configuration:

   # The monitoring stack doesn't currently have alert rules configured
   # You would need to add them to the prometheus_config in:
   vim /home/homelab/organized/scripts/homelab/hosts/vms/homelab-vm/monitoring.yaml
   

2. Add Alert Rules Section:

   rule_files:
     - "/etc/prometheus/alert-rules.yml"
   
   alerting:
     alertmanagers:
       - static_configs:
           - targets:
             - alertmanager:9093
   

3. Create Alert Rules Config:

   # Add to configs section in monitoring.yaml
   alert_rules:
     content: |
       groups:
         - name: homelab-alerts
           rules:
             - alert: HighCPUUsage
               expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
               for: 5m
               labels:
                 severity: warning
               annotations:
                 summary: "High CPU usage on {{ $labels.instance }}"
                 description: "CPU usage is above 80% for 5 minutes"
   

Modifying Alert Severity and Routing

File: hosts/vms/homelab-vm/alerting.yaml

1. Change Alert Routing:

   # Lines 30-37: Modify routing rules
   routes:
     - match:
         severity: critical
       receiver: 'critical-alerts'
     - match:
         severity: warning
       receiver: 'ntfy-all'
   

2. Add New Receivers:

   # Lines 39-50: Add new notification channels
   receivers:
     - name: 'email-alerts'
       email_configs:
         - to: 'admin@yourdomain.com'
           subject: 'Homelab Alert: {{ .GroupLabels.alertname }}'
   

Customizing Notification Format

File: hosts/vms/homelab-vm/alerting.yaml (lines 85-109)

The format_alert() function controls how notifications appear:

def format_alert(alert):
    # Customize title format
    title = f"{alertname} [{status_text}] - {instance}"
    
    # Customize message body
    body_parts = []
    if summary:
        body_parts.append(f"📊 {summary}")
    if description:
        body_parts.append(f"📝 {description}")
    
    # Add custom fields
    body_parts.append(f"🕐 {datetime.now().strftime('%H:%M:%S')}")
    
    return title, body, severity, status

---

📱 Mobile App Setup

iOS Setup

1. Install ntfy app from the App Store
2. Add subscription:
   • Server: https://ntfy.vish.gg
   • Topic: homelab-alerts
3. Enable notifications in iOS Settings
4. Important: The server must have upstream-base-url: "https://ntfy.sh" configured for iOS push notifications to work

Android Setup

1. Install ntfy app from Google Play Store or F-Droid
2. Add subscription:
   • Server: https://ntfy.vish.gg
   • Topic: homelab-alerts
3. Configure notification settings as desired

Web Interface

Access the web interface at:

• Internal: http://atlantis.vish.local:8081
• External: https://ntfy.vish.gg

---

🧪 Testing Notifications

Test Scripts Available

Location: /home/homelab/organized/scripts/homelab/scripts/test-ntfy-notifications.sh

Manual Testing

1. Test Direct ntfy:

   curl -H "Title: Test Alert" -d "This is a test notification" https://ntfy.vish.gg/REDACTED_NTFY_TOPIC
   

2. Test Alert Bridge:

   curl -X POST http://atlantis.vish.local:5001/alert -H "Content-Type: application/json" -d '{
     "alerts": [{
       "status": "firing",
       "labels": {"alertname": "TestAlert", "severity": "warning", "instance": "test:9100"},
       "annotations": {"summary": "Test alert", "description": "This is a test notification"}
     }]
   }'
   

3. Test Signal Bridge (for critical alerts):

   curl -X POST http://atlantis.vish.local:5000/alert -H "Content-Type: application/json" -d '{
     "alerts": [{
       "status": "firing",
       "labels": {"alertname": "TestAlert", "severity": "critical", "instance": "test:9100"},
       "annotations": {"summary": "Critical test alert", "description": "This is a critical test"}
     }]
   }'
   

4. Test Gitea Bridge:

   curl -X POST http://atlantis.vish.local:8095 -H "X-Gitea-Event: push" -H "Content-Type: application/json" -d '{
     "repository": {"full_name": "test/repo"},
     "sender": {"login": "testuser"},
     "commits": [{"message": "Test commit"}],
     "ref": "refs/heads/main"
   }'
   

---

🔍 Troubleshooting

Common Issues

1. Notifications not received on iOS:

   • Verify upstream-base-url: "https://ntfy.sh" is set in server config
   • Restart ntfy container: docker restart NTFY
   • Re-subscribe in iOS app

2. Alerts not firing:

   • Check Prometheus targets: http://atlantis.vish.local:9090/targets
   • Check Alertmanager: http://atlantis.vish.local:9093
   • Verify bridge health: curl http://atlantis.vish.local:5001/health

3. Signal notifications not working:

   • Check signal-api container: docker logs signal-api
   • Test signal-bridge: curl http://atlantis.vish.local:5000/health

Container Status Check

# Via Portainer API
curl -s -H "X-API-Key: "REDACTED_API_KEY" \
  "http://atlantis.vish.local:10000/api/endpoints/443399/docker/containers/json" | \
  jq '.[] | select(.Names[0] | contains("ntfy") or contains("alert")) | {Names: .Names, State: .State, Status: .Status}'

Log Access

• ntfy logs: Check via Portainer → Containers → NTFY → Logs
• Bridge logs: Check via Portainer → Containers → ntfy-bridge → Logs
• Alertmanager logs: Check via Portainer → Containers → alertmanager → Logs

---

📊 Current Deployment Status

Portainer Stacks

Stack Name	Status	Endpoint	Configuration File
ntfy-stack	✅ Running	homelab-vm (443399)	hosts/vms/homelab-vm/ntfy.yaml
alerting-stack	✅ Running	homelab-vm (443399)	hosts/vms/homelab-vm/alerting.yaml
monitoring-stack	✅ Running	homelab-vm (443399)	hosts/vms/homelab-vm/monitoring.yaml
signal-api-stack	✅ Running	homelab-vm (443399)	hosts/vms/homelab-vm/signal_api.yaml

Container Health

Container	Image	Status	Purpose
NTFY	binwiederhier/ntfy	✅ Running	Main notification server
alertmanager	prom/alertmanager:latest	✅ Running	Alert routing
ntfy-bridge	python:3.11-slim	✅ Running (healthy)	Alert formatting
signal-bridge	python:3.11-slim	✅ Running (healthy)	Signal forwarding
gitea-ntfy-bridge	python:3.12-alpine	✅ Running	Git notifications
prometheus	prom/prometheus:latest	✅ Running	Metrics collection
grafana	grafana/grafana-oss:latest	✅ Running	Monitoring dashboard

---

🔐 Security Considerations

1. ntfy Server: Publicly accessible at https://ntfy.vish.gg
2. Topic Security: Uses a single topic homelab-alerts - consider authentication if needed
3. Signal Integration: Uses encrypted Signal messaging for critical alerts
4. Internal Network: Most bridges communicate over internal Docker networks

---

📚 Additional Resources

• ntfy Documentation: https://ntfy.sh/REDACTED_TOPIC/
• Alertmanager Documentation: https://prometheus.io/docs/alerting/latest/alertmanager/
• Prometheus Alerting: https://prometheus.io/docs/alerting/latest/rules/

---

🔄 Maintenance Tasks

Regular Maintenance

1. Monthly: Check container health and logs
2. Quarterly: Test all notification channels
3. As needed: Update notification rules based on infrastructure changes

Backup Important Configs

# Backup ntfy configuration
cp /home/homelab/docker/ntfy/config/server.yml /backup/ntfy-config-$(date +%Y%m%d).yml

# Backup alerting configuration (already in Git)
git -C /home/homelab/organized/scripts/homelab status

---

This documentation reflects the current state of your ntfy notification system as of January 2025. For the most up-to-date configuration, always refer to the actual configuration files in the homelab Git repository.