# Homelab Alerting Stack

This adds Prometheus Alertmanager with notifications to both **ntfy** and **Signal**.

## Components

| Component | Purpose | Port |
|-----------|---------|------|
| Alertmanager | Routes alerts based on severity | 9093 |
| Signal Bridge | Forwards critical alerts to Signal | 5000 |

## Alert Routing

- **Warning alerts** → ntfy only (`homelab-alerts` topic)
- **Critical alerts** → Both ntfy AND Signal

## Deployment Steps

### 1. Update your phone number

Edit `docker-compose.alerting.yml` and replace `REPLACE_WITH_YOUR_NUMBER`:

```yaml
environment:
  - SIGNAL_SENDER=+REDACTED_PHONE_NUMBER      # Your Signal number
  - SIGNAL_RECIPIENTS=+REDACTED_PHONE_NUMBER  # Where to send alerts
```

### 2. Copy files to Homelab VM

```bash
# On your local machine or wherever you have SSH access
scp -r alerting-configs/* homelab@192.168.0.210:~/docker/monitoring/
```

### 3. Update Prometheus config

Replace the existing `prometheus.yml` with `prometheus-updated.yml`:

```bash
cd ~/docker/monitoring
cp prometheus-updated.yml prometheus/prometheus.yml
cp alert-rules.yml prometheus/alert-rules.yml
```

### 4. Create alertmanager directory

```bash
mkdir -p alertmanager
cp alertmanager.yml alertmanager/
```

### 5. Deploy the alerting stack

```bash
# Build and start alertmanager + signal bridge
docker-compose -f docker-compose.alerting.yml up -d --build

# Reload Prometheus to pick up new config
curl -X POST http://localhost:9090/-/reload
```

### 6. Verify deployment

```bash
# Check Alertmanager is running
curl http://localhost:9093/-/healthy

# Check Signal Bridge is running  
curl http://localhost:5000/health

# Send test alert to Signal
curl -X POST http://localhost:5000/test \
  -H "Content-Type: application/json" \
  -d '{"message": "🧪 Test alert from Homelab!"}'

# Send test notification to ntfy
curl -d "Test alert from Alertmanager setup" https://ntfy.vish.gg/REDACTED_NTFY_TOPIC
```

## Alert Rules Included

| Alert | Severity | Trigger |
|-------|----------|---------|
| HostDown | Critical | Host unreachable for 2 min |
| REDACTED_APP_PASSWORD | Warning | CPU > 80% for 5 min |
| HostCriticalCpuUsage | Critical | CPU > 95% for 5 min |
| HostHighMemoryUsage | Warning | Memory > 85% for 5 min |
| HostCriticalMemoryUsage | Critical | Memory > 95% for 5 min |
| HostOutOfMemory | Critical | Memory < 5% available |
| HostHighDiskUsage | Warning | Disk > 80% full |
| HostCriticalDiskUsage | Critical | Disk > 90% full |
| HostDiskWillFillIn24Hours | Warning | Predicted to fill in 24h |
| REDACTED_APP_PASSWORD | Critical | Filesystem became read-only |
| HostNetworkErrors | Warning | Network errors detected |
| HostClockSkew | Warning | Time drift > 0.5 seconds |

## Receiving Alerts

### ntfy App
1. Install ntfy app on your phone (iOS/Android)
2. Add server: `https://ntfy.vish.gg`
3. Subscribe to topic: `homelab-alerts`

### Signal
- Alerts will arrive as regular Signal messages from your registered number

## Troubleshooting

### Check Alertmanager status
```bash
docker logs alertmanager
curl http://localhost:9093/api/v2/status
```

### Check active alerts
```bash
curl http://localhost:9093/api/v2/alerts
```

### Check Signal Bridge logs
```bash
docker logs signal-bridge
```

### Manually trigger test alert in Prometheus
Add this rule temporarily to test:
```yaml
- alert: TestAlert
  expr: vector(1)
  labels:
    severity: warning
  annotations:
    summary: "Test alert"
```