Sanitized mirror from private repository - 2026-03-17 11:52:42 UTC

docs/admin/monitoring-setup.md

@@ -0,0 +1,129 @@
+# Homelab Monitoring & Alerting Setup
+
+## Overview
+
+| Service | Host | Port | URL | Purpose |
+|---------|------|------|-----|---------|
+| **Grafana** | Homelab VM | 3300 | `https://gf.vish.gg` | Dashboards & alerting |
+| **Prometheus** | Homelab VM | 9090 | `http://192.168.0.210:9090` | Metrics collection |
+| **Ntfy** | Homelab VM | 8081 | `http://192.168.0.210:8081` | Push notifications |
+| **Uptime Kuma** | rpi5 | 3001 | `http://<rpi5-ip>:3001` | Uptime monitoring |
+
+All services are deployed as `monitoring-stack` (Portainer stack ID 687) via GitOps from `hosts/vms/homelab-vm/monitoring.yaml`.
+
+### Grafana Details
+- **Version**: 12.4.0 (pinned)
+- **Login**: Authentik SSO at `https://gf.vish.gg` (primary) or local `admin` account
+- **Default home dashboard**: Node Details - Full Metrics (`node-details-v2`)
+- **Dashboards**: Infrastructure Overview, Node Details, Synology NAS, Node Exporter Full
+
+## Ntfy Setup
+
+### Access
+- Web UI: `http://192.168.0.210:8081`
+- Subscribe to topics: `http://192.168.0.210:8081/<topic>`
+
+### Recommended Topics
+- `alerts` - Critical system alerts
+- `homelab` - General notifications  
+- `updates` - Container update notifications
+
+### Mobile App Setup
+1. Install Ntfy app (Android/iOS)
+2. Add server: `http://192.168.0.210:8081` (or your public URL)
+3. Subscribe to topics: `alerts`, `homelab`
+
+### Test Notification
+```bash
+curl -d "Test notification from homelab" http://192.168.0.210:8081/alerts
+```
+
+---
+
+## Grafana Alerting
+
+### Access
+- External: `https://gf.vish.gg` (Authentik SSO)
+- Internal: `http://192.168.0.210:3300`
+
+### Configure Ntfy Contact Point
+1. Go to: Alerting → Contact Points → Add
+2. Name: `Ntfy`
+3. Type: `Webhook`
+4. URL: `http://NTFY:80/alerts` (internal) or `http://192.168.0.210:8081/alerts`
+5. HTTP Method: `POST`
+
+### Configure Email Contact Point (SMTP)
+1. Edit `grafana.ini` or use environment variables:
+```ini
+[smtp]
+enabled = true
+host = smtp.gmail.com:587
+user = your-email@gmail.com
+password = "REDACTED_PASSWORD"
+from_address = your-email@gmail.com
+```
+
+### Sample Alert Rules
+- CPU > 80% for 5 minutes
+- Memory > 90% for 5 minutes
+- Disk > 85%
+- Container down
+
+---
+
+## Uptime Kuma Notifications
+
+### Access
+- URL: `http://<rpi5-ip>:3001`
+
+### Add Ntfy Notification
+1. Settings → Notifications → Setup Notification
+2. Type: `Ntfy`
+3. Server URL: `http://192.168.0.210:8081`
+4. Topic: `alerts`
+5. Priority: `high` (for critical) or `default`
+
+### Add Email Notification (SMTP)
+1. Settings → Notifications → Setup Notification
+2. Type: `SMTP`
+3. Host: `smtp.gmail.com`
+4. Port: `587`
+5. Security: `STARTTLS`
+6. Username: your email
+7. Password: app password
+8. From: your email
+9. To: recipient email
+
+### Recommended Monitors
+- All Portainer endpoints (HTTP)
+- Key services: Gitea, Plex, Grafana, etc.
+- External services you depend on
+
+---
+
+## Watchtower Notifications
+
+Watchtower can notify on container updates. Add to compose:
+
+```yaml
+environment:
+  - WATCHTOWER_NOTIFICATIONS=shoutrrr
+  - WATCHTOWER_NOTIFICATION_URL=ntfy://192.168.0.210:8081/updates
+```
+
+---
+
+## Quick Test Commands
+
+```bash
+# Test Ntfy
+curl -d "🔔 Test alert" http://192.168.0.210:8081/alerts
+
+# Test with priority
+curl -H "Priority: high" -H "Title: Critical Alert" \
+  -d "Something needs attention!" http://192.168.0.210:8081/alerts
+
+# Test with tags/emoji
+curl -H "Tags: warning,server" -d "Server alert" http://192.168.0.210:8081/homelab
+```