homelab-optimized/archive/deprecated-monitoring-stacks/prometheus_grafana_hub/alerting

Homelab Alerting Stack

This adds Prometheus Alertmanager with notifications to both ntfy and Signal.

Components

Component	Purpose	Port
Alertmanager	Routes alerts based on severity	9093
Signal Bridge	Forwards critical alerts to Signal	5000

Alert Routing

• Warning alerts → ntfy only (homelab-alerts topic)
• Critical alerts → Both ntfy AND Signal

Deployment Steps

1. Update your phone number

Edit docker-compose.alerting.yml and replace REPLACE_WITH_YOUR_NUMBER:

environment:
  - SIGNAL_SENDER=+REDACTED_PHONE_NUMBER      # Your Signal number
  - SIGNAL_RECIPIENTS=+REDACTED_PHONE_NUMBER  # Where to send alerts

2. Copy files to Homelab VM

# On your local machine or wherever you have SSH access
scp -r alerting-configs/* homelab@192.168.0.210:~/docker/monitoring/

3. Update Prometheus config

Replace the existing prometheus.yml with prometheus-updated.yml:

cd ~/docker/monitoring
cp prometheus-updated.yml prometheus/prometheus.yml
cp alert-rules.yml prometheus/alert-rules.yml

4. Create alertmanager directory

mkdir -p alertmanager
cp alertmanager.yml alertmanager/

5. Deploy the alerting stack

# Build and start alertmanager + signal bridge
docker-compose -f docker-compose.alerting.yml up -d --build

# Reload Prometheus to pick up new config
curl -X POST http://localhost:9090/-/reload

6. Verify deployment

# Check Alertmanager is running
curl http://localhost:9093/-/healthy

# Check Signal Bridge is running  
curl http://localhost:5000/health

# Send test alert to Signal
curl -X POST http://localhost:5000/test \
  -H "Content-Type: application/json" \
  -d '{"message": "🧪 Test alert from Homelab!"}'

# Send test notification to ntfy
curl -d "Test alert from Alertmanager setup" https://ntfy.vish.gg/REDACTED_NTFY_TOPIC

Alert Rules Included

Alert	Severity	Trigger
HostDown	Critical	Host unreachable for 2 min
REDACTED_APP_PASSWORD	Warning	CPU > 80% for 5 min
HostCriticalCpuUsage	Critical	CPU > 95% for 5 min
HostHighMemoryUsage	Warning	Memory > 85% for 5 min
HostCriticalMemoryUsage	Critical	Memory > 95% for 5 min
HostOutOfMemory	Critical	Memory < 5% available
HostHighDiskUsage	Warning	Disk > 80% full
HostCriticalDiskUsage	Critical	Disk > 90% full
HostDiskWillFillIn24Hours	Warning	Predicted to fill in 24h
REDACTED_APP_PASSWORD	Critical	Filesystem became read-only
HostNetworkErrors	Warning	Network errors detected
HostClockSkew	Warning	Time drift > 0.5 seconds

Receiving Alerts

ntfy App

1. Install ntfy app on your phone (iOS/Android)
2. Add server: https://ntfy.vish.gg
3. Subscribe to topic: homelab-alerts

Signal

• Alerts will arrive as regular Signal messages from your registered number

Troubleshooting

Check Alertmanager status

docker logs alertmanager
curl http://localhost:9093/api/v2/status

Check active alerts

curl http://localhost:9093/api/v2/alerts

Check Signal Bridge logs

docker logs signal-bridge

Manually trigger test alert in Prometheus

Add this rule temporarily to test:

- alert: TestAlert
  expr: vector(1)
  labels:
    severity: warning
  annotations:
    summary: "Test alert"