Sanitized mirror from private repository - 2026-04-19 08:15:48 UTC

archive/deprecated-monitoring-stacks/prometheus_grafana_hub/alerting/README.md

@@ -0,0 +1,135 @@
+# Homelab Alerting Stack
+
+This adds Prometheus Alertmanager with notifications to both **ntfy** and **Signal**.
+
+## Components
+
+| Component | Purpose | Port |
+|-----------|---------|------|
+| Alertmanager | Routes alerts based on severity | 9093 |
+| Signal Bridge | Forwards critical alerts to Signal | 5000 |
+
+## Alert Routing
+
+- **Warning alerts** → ntfy only (`homelab-alerts` topic)
+- **Critical alerts** → Both ntfy AND Signal
+
+## Deployment Steps
+
+### 1. Update your phone number
+
+Edit `docker-compose.alerting.yml` and replace `REPLACE_WITH_YOUR_NUMBER`:
+
+```yaml
+environment:
+  - SIGNAL_SENDER=+REDACTED_PHONE_NUMBER      # Your Signal number
+  - SIGNAL_RECIPIENTS=+REDACTED_PHONE_NUMBER  # Where to send alerts
+```
+
+### 2. Copy files to Homelab VM
+
+```bash
+# On your local machine or wherever you have SSH access
+scp -r alerting-configs/* homelab@192.168.0.210:~/docker/monitoring/
+```
+
+### 3. Update Prometheus config
+
+Replace the existing `prometheus.yml` with `prometheus-updated.yml`:
+
+```bash
+cd ~/docker/monitoring
+cp prometheus-updated.yml prometheus/prometheus.yml
+cp alert-rules.yml prometheus/alert-rules.yml
+```
+
+### 4. Create alertmanager directory
+
+```bash
+mkdir -p alertmanager
+cp alertmanager.yml alertmanager/
+```
+
+### 5. Deploy the alerting stack
+
+```bash
+# Build and start alertmanager + signal bridge
+docker-compose -f docker-compose.alerting.yml up -d --build
+
+# Reload Prometheus to pick up new config
+curl -X POST http://localhost:9090/-/reload
+```
+
+### 6. Verify deployment
+
+```bash
+# Check Alertmanager is running
+curl http://localhost:9093/-/healthy
+
+# Check Signal Bridge is running  
+curl http://localhost:5000/health
+
+# Send test alert to Signal
+curl -X POST http://localhost:5000/test \
+  -H "Content-Type: application/json" \
+  -d '{"message": "🧪 Test alert from Homelab!"}'
+
+# Send test notification to ntfy
+curl -d "Test alert from Alertmanager setup" https://ntfy.vish.gg/REDACTED_NTFY_TOPIC
+```
+
+## Alert Rules Included
+
+| Alert | Severity | Trigger |
+|-------|----------|---------|
+| HostDown | Critical | Host unreachable for 2 min |
+| REDACTED_APP_PASSWORD | Warning | CPU > 80% for 5 min |
+| HostCriticalCpuUsage | Critical | CPU > 95% for 5 min |
+| HostHighMemoryUsage | Warning | Memory > 85% for 5 min |
+| HostCriticalMemoryUsage | Critical | Memory > 95% for 5 min |
+| HostOutOfMemory | Critical | Memory < 5% available |
+| HostHighDiskUsage | Warning | Disk > 80% full |
+| HostCriticalDiskUsage | Critical | Disk > 90% full |
+| HostDiskWillFillIn24Hours | Warning | Predicted to fill in 24h |
+| REDACTED_APP_PASSWORD | Critical | Filesystem became read-only |
+| HostNetworkErrors | Warning | Network errors detected |
+| HostClockSkew | Warning | Time drift > 0.5 seconds |
+
+## Receiving Alerts
+
+### ntfy App
+1. Install ntfy app on your phone (iOS/Android)
+2. Add server: `https://ntfy.vish.gg`
+3. Subscribe to topic: `homelab-alerts`
+
+### Signal
+- Alerts will arrive as regular Signal messages from your registered number
+
+## Troubleshooting
+
+### Check Alertmanager status
+```bash
+docker logs alertmanager
+curl http://localhost:9093/api/v2/status
+```
+
+### Check active alerts
+```bash
+curl http://localhost:9093/api/v2/alerts
+```
+
+### Check Signal Bridge logs
+```bash
+docker logs signal-bridge
+```
+
+### Manually trigger test alert in Prometheus
+Add this rule temporarily to test:
+```yaml
+- alert: TestAlert
+  expr: vector(1)
+  labels:
+    severity: warning
+  annotations:
+    summary: "Test alert"
+```