# Homelab Monitoring & Alerting Setup

## Overview

| Service | Host | Port | URL | Purpose |
|---------|------|------|-----|---------|
| **Grafana** | Homelab VM | 3300 | `https://gf.vish.gg` | Dashboards & alerting |
| **Prometheus** | Homelab VM | 9090 | `http://192.168.0.210:9090` | Metrics collection |
| **Ntfy** | Homelab VM | 8081 | `http://192.168.0.210:8081` | Push notifications |
| **Uptime Kuma** | rpi5 | 3001 | `http://<rpi5-ip>:3001` | Uptime monitoring |

All services are deployed as `monitoring-stack` (Portainer stack ID 687) via GitOps from `hosts/vms/homelab-vm/monitoring.yaml`.

### Grafana Details
- **Version**: 12.4.0 (pinned)
- **Login**: Authentik SSO at `https://gf.vish.gg` (primary) or local `admin` account
- **Default home dashboard**: Node Details - Full Metrics (`node-details-v2`)
- **Dashboards**: Infrastructure Overview, Node Details, Synology NAS, Node Exporter Full

## Ntfy Setup

### Access
- Web UI: `http://192.168.0.210:8081`
- Subscribe to topics: `http://192.168.0.210:8081/<topic>`

### Recommended Topics
- `alerts` - Critical system alerts
- `homelab` - General notifications  
- `updates` - Container update notifications

### Mobile App Setup
1. Install Ntfy app (Android/iOS)
2. Add server: `http://192.168.0.210:8081` (or your public URL)
3. Subscribe to topics: `alerts`, `homelab`

### Test Notification
```bash
curl -d "Test notification from homelab" http://192.168.0.210:8081/alerts
```

---

## Grafana Alerting

### Access
- External: `https://gf.vish.gg` (Authentik SSO)
- Internal: `http://192.168.0.210:3300`

### Configure Ntfy Contact Point
1. Go to: Alerting → Contact Points → Add
2. Name: `Ntfy`
3. Type: `Webhook`
4. URL: `http://NTFY:80/alerts` (internal) or `http://192.168.0.210:8081/alerts`
5. HTTP Method: `POST`

### Configure Email Contact Point (SMTP)
1. Edit `grafana.ini` or use environment variables:
```ini
[smtp]
enabled = true
host = smtp.gmail.com:587
user = your-email@gmail.com
password = "REDACTED_PASSWORD"
from_address = your-email@gmail.com
```

### Sample Alert Rules
- CPU > 80% for 5 minutes
- Memory > 90% for 5 minutes
- Disk > 85%
- Container down

---

## Uptime Kuma Notifications

### Access
- URL: `http://<rpi5-ip>:3001`

### Add Ntfy Notification
1. Settings → Notifications → Setup Notification
2. Type: `Ntfy`
3. Server URL: `http://192.168.0.210:8081`
4. Topic: `alerts`
5. Priority: `high` (for critical) or `default`

### Add Email Notification (SMTP)
1. Settings → Notifications → Setup Notification
2. Type: `SMTP`
3. Host: `smtp.gmail.com`
4. Port: `587`
5. Security: `STARTTLS`
6. Username: your email
7. Password: app password
8. From: your email
9. To: recipient email

### Recommended Monitors
- All Portainer endpoints (HTTP)
- Key services: Gitea, Plex, Grafana, etc.
- External services you depend on

---

## Watchtower Notifications

Watchtower can notify on container updates. Add to compose:

```yaml
environment:
  - WATCHTOWER_NOTIFICATIONS=shoutrrr
  - WATCHTOWER_NOTIFICATION_URL=ntfy://192.168.0.210:8081/updates
```

---

## Quick Test Commands

```bash
# Test Ntfy
curl -d "🔔 Test alert" http://192.168.0.210:8081/alerts

# Test with priority
curl -H "Priority: high" -H "Title: Critical Alert" \
  -d "Something needs attention!" http://192.168.0.210:8081/alerts

# Test with tags/emoji
curl -H "Tags: warning,server" -d "Server alert" http://192.168.0.210:8081/homelab
```