130 lines
3.3 KiB
Markdown
130 lines
3.3 KiB
Markdown
# Homelab Monitoring & Alerting Setup
|
|
|
|
## Overview
|
|
|
|
| Service | Host | Port | URL | Purpose |
|
|
|---------|------|------|-----|---------|
|
|
| **Grafana** | Homelab VM | 3300 | `https://gf.vish.gg` | Dashboards & alerting |
|
|
| **Prometheus** | Homelab VM | 9090 | `http://192.168.0.210:9090` | Metrics collection |
|
|
| **Ntfy** | Homelab VM | 8081 | `http://192.168.0.210:8081` | Push notifications |
|
|
| **Uptime Kuma** | rpi5 | 3001 | `http://<rpi5-ip>:3001` | Uptime monitoring |
|
|
|
|
All services are deployed as `monitoring-stack` (Portainer stack ID 687) via GitOps from `hosts/vms/homelab-vm/monitoring.yaml`.
|
|
|
|
### Grafana Details
|
|
- **Version**: 12.4.0 (pinned)
|
|
- **Login**: Authentik SSO at `https://gf.vish.gg` (primary) or local `admin` account
|
|
- **Default home dashboard**: Node Details - Full Metrics (`node-details-v2`)
|
|
- **Dashboards**: Infrastructure Overview, Node Details, Synology NAS, Node Exporter Full
|
|
|
|
## Ntfy Setup
|
|
|
|
### Access
|
|
- Web UI: `http://192.168.0.210:8081`
|
|
- Subscribe to topics: `http://192.168.0.210:8081/<topic>`
|
|
|
|
### Recommended Topics
|
|
- `alerts` - Critical system alerts
|
|
- `homelab` - General notifications
|
|
- `updates` - Container update notifications
|
|
|
|
### Mobile App Setup
|
|
1. Install Ntfy app (Android/iOS)
|
|
2. Add server: `http://192.168.0.210:8081` (or your public URL)
|
|
3. Subscribe to topics: `alerts`, `homelab`
|
|
|
|
### Test Notification
|
|
```bash
|
|
curl -d "Test notification from homelab" http://192.168.0.210:8081/alerts
|
|
```
|
|
|
|
---
|
|
|
|
## Grafana Alerting
|
|
|
|
### Access
|
|
- External: `https://gf.vish.gg` (Authentik SSO)
|
|
- Internal: `http://192.168.0.210:3300`
|
|
|
|
### Configure Ntfy Contact Point
|
|
1. Go to: Alerting → Contact Points → Add
|
|
2. Name: `Ntfy`
|
|
3. Type: `Webhook`
|
|
4. URL: `http://NTFY:80/alerts` (internal) or `http://192.168.0.210:8081/alerts`
|
|
5. HTTP Method: `POST`
|
|
|
|
### Configure Email Contact Point (SMTP)
|
|
1. Edit `grafana.ini` or use environment variables:
|
|
```ini
|
|
[smtp]
|
|
enabled = true
|
|
host = smtp.gmail.com:587
|
|
user = your-email@gmail.com
|
|
password = "REDACTED_PASSWORD"
|
|
from_address = your-email@gmail.com
|
|
```
|
|
|
|
### Sample Alert Rules
|
|
- CPU > 80% for 5 minutes
|
|
- Memory > 90% for 5 minutes
|
|
- Disk > 85%
|
|
- Container down
|
|
|
|
---
|
|
|
|
## Uptime Kuma Notifications
|
|
|
|
### Access
|
|
- URL: `http://<rpi5-ip>:3001`
|
|
|
|
### Add Ntfy Notification
|
|
1. Settings → Notifications → Setup Notification
|
|
2. Type: `Ntfy`
|
|
3. Server URL: `http://192.168.0.210:8081`
|
|
4. Topic: `alerts`
|
|
5. Priority: `high` (for critical) or `default`
|
|
|
|
### Add Email Notification (SMTP)
|
|
1. Settings → Notifications → Setup Notification
|
|
2. Type: `SMTP`
|
|
3. Host: `smtp.gmail.com`
|
|
4. Port: `587`
|
|
5. Security: `STARTTLS`
|
|
6. Username: your email
|
|
7. Password: app password
|
|
8. From: your email
|
|
9. To: recipient email
|
|
|
|
### Recommended Monitors
|
|
- All Portainer endpoints (HTTP)
|
|
- Key services: Gitea, Plex, Grafana, etc.
|
|
- External services you depend on
|
|
|
|
---
|
|
|
|
## Watchtower Notifications
|
|
|
|
Watchtower can notify on container updates. Add to compose:
|
|
|
|
```yaml
|
|
environment:
|
|
- WATCHTOWER_NOTIFICATIONS=shoutrrr
|
|
- WATCHTOWER_NOTIFICATION_URL=ntfy://192.168.0.210:8081/updates
|
|
```
|
|
|
|
---
|
|
|
|
## Quick Test Commands
|
|
|
|
```bash
|
|
# Test Ntfy
|
|
curl -d "🔔 Test alert" http://192.168.0.210:8081/alerts
|
|
|
|
# Test with priority
|
|
curl -H "Priority: high" -H "Title: Critical Alert" \
|
|
-d "Something needs attention!" http://192.168.0.210:8081/alerts
|
|
|
|
# Test with tags/emoji
|
|
curl -H "Tags: warning,server" -d "Server alert" http://192.168.0.210:8081/homelab
|
|
```
|