144 lines
5.3 KiB
Markdown
144 lines
5.3 KiB
Markdown
# Tailscale Host Monitoring Status Report
|
|
|
|
> **⚠️ Historical Snapshot**: This document was generated on Feb 15, 2026. The alerts and offline status listed here are no longer current. For live node status, run `tailscale status` on the homelab VM or check Grafana at `http://100.67.40.126:3000`.
|
|
|
|
## 📊 Status Snapshot
|
|
|
|
**Generated:** February 15, 2026
|
|
|
|
### Monitored Tailscale Hosts (13 total)
|
|
|
|
#### ✅ Online Hosts (10)
|
|
- **atlantis-node** (100.83.230.112:9100) - Synology NAS
|
|
- **atlantis-snmp** (100.83.230.112) - SNMP monitoring
|
|
- **calypso-node** (100.103.48.78:9100) - Node exporter
|
|
- **calypso-snmp** (100.103.48.78) - SNMP monitoring
|
|
- **concord-nuc-node** (100.72.55.21:9100) - Intel NUC
|
|
- **proxmox-node** (100.87.12.28:9100) - Proxmox server
|
|
- **raspberry-pis** (100.77.151.40:9100) - Pi cluster node
|
|
- **setillo-node** (100.125.0.20:9100) - Node exporter
|
|
- **setillo-snmp** (100.125.0.20) - SNMP monitoring
|
|
- **truenas-node** (100.75.252.64:9100) - TrueNAS server
|
|
|
|
#### ❌ Offline Hosts (3)
|
|
- **homelab-node** (100.67.40.126:9100) - Main homelab VM
|
|
- **raspberry-pis** (100.123.246.75:9100) - Pi cluster node
|
|
- **vmi2076105-node** (100.99.156.20:9100) - VPS instance
|
|
|
|
## 🚨 Active Alerts
|
|
|
|
### Critical HostDown Alerts (2 firing)
|
|
1. **vmi2076105-node** (100.99.156.20:9100)
|
|
- Status: Firing since Feb 14, 07:57 UTC
|
|
- Duration: ~24 hours
|
|
- Notifications: Sent to ntfy + Signal
|
|
|
|
2. **homelab-node** (100.67.40.126:9100)
|
|
- Status: Firing since Feb 14, 09:23 UTC
|
|
- Duration: ~22 hours
|
|
- Notifications: Sent to ntfy + Signal
|
|
|
|
## 📬 Notification System Status
|
|
|
|
### ✅ Working Notification Channels
|
|
- **ntfy**: http://192.168.0.210:8081/homelab-alerts ✅
|
|
- **Signal**: Via signal-bridge (critical alerts) ✅
|
|
- **Alertmanager**: http://100.67.40.126:9093 ✅
|
|
|
|
### Test Results
|
|
- ntfy notification test: **PASSED** ✅
|
|
- Message delivery: **CONFIRMED** ✅
|
|
- Alert routing: **WORKING** ✅
|
|
|
|
## ⚙️ Monitoring Configuration
|
|
|
|
### Alert Rules
|
|
- **Trigger**: Host unreachable for 2+ minutes
|
|
- **Severity**: Critical (dual-channel notifications)
|
|
- **Query**: `up{job=~".*-node"} == 0`
|
|
- **Evaluation**: Every 30 seconds
|
|
|
|
### Notification Routing
|
|
- **Warning alerts** → ntfy only
|
|
- **Critical alerts** → ntfy + Signal
|
|
- **Resolved alerts** → Both channels
|
|
|
|
## 🔧 Infrastructure Details
|
|
|
|
### Monitoring Stack
|
|
- **Prometheus**: http://100.67.40.126:9090
|
|
- **Grafana**: http://100.67.40.126:3000
|
|
- **Alertmanager**: http://100.67.40.126:9093
|
|
- **Bridge Services**: ntfy-bridge (5001), signal-bridge (5000)
|
|
|
|
### Data Collection
|
|
- **Node Exporter**: System metrics on port 9100
|
|
- **SNMP Exporter**: Network device metrics on port 9116
|
|
- **Scrape Interval**: 15 seconds
|
|
- **Retention**: Default Prometheus retention
|
|
|
|
## 📋 Recommendations
|
|
|
|
### Immediate Actions
|
|
1. **Investigate offline hosts**:
|
|
- Check homelab-node (100.67.40.126) - main VM down
|
|
- Verify vmi2076105-node (100.99.156.20) - VPS status
|
|
- Check raspberry-pis node (100.123.246.75)
|
|
|
|
2. **Verify notifications**:
|
|
- Confirm you're receiving ntfy alerts on mobile
|
|
- Test Signal notifications for critical alerts
|
|
|
|
### Maintenance
|
|
- Monitor disk space on active hosts
|
|
- Review alert thresholds if needed
|
|
- Consider adding more monitoring targets
|
|
|
|
## 🧪 Testing
|
|
|
|
Use the test script to verify monitoring:
|
|
```bash
|
|
./scripts/test-tailscale-monitoring.sh
|
|
```
|
|
|
|
For manual testing:
|
|
1. Stop node_exporter on any host: `sudo systemctl stop node_exporter`
|
|
2. Wait 2+ minutes for alert to fire
|
|
3. Check ntfy app and Signal for notifications
|
|
4. Restart: `sudo systemctl start node_exporter`
|
|
|
|
---
|
|
|
|
## 🟢 Verified Online Nodes (March 2026)
|
|
|
|
As of March 11, 2026, all 16 active nodes verified reachable via ping:
|
|
|
|
| Node | Tailscale IP | Role |
|
|
|------|-------------|------|
|
|
| atlantis | 100.83.230.112 | Primary NAS, exit node |
|
|
| calypso | 100.103.48.78 | Secondary NAS, Headscale host |
|
|
| setillo | 100.125.0.20 | Remote NAS, Tucson |
|
|
| homelab | 100.67.40.126 | Main VM (this host) |
|
|
| pve | 100.87.12.28 | Proxmox hypervisor |
|
|
| vish-concord-nuc | 100.72.55.21 | Intel NUC, exit node |
|
|
| pi-5 | 100.77.151.40 | Raspberry Pi 5 |
|
|
| matrix-ubuntu | 100.85.21.51 | Atlantis VM |
|
|
| guava | 100.75.252.64 | TrueNAS Scale |
|
|
| jellyfish | 100.69.121.120 | Pi 5 media/NAS |
|
|
| gl-mt3000 | 100.126.243.15 | GL.iNet router (remote), SSH alias `gl-mt3000` |
|
|
| gl-be3600 | 100.105.59.123 | GL.iNet router (Concord), exit node |
|
|
| homeassistant | 100.112.186.90 | HA Green (via GL-MT3000 subnet) |
|
|
| seattle | 100.82.197.124 | Contabo VPS, exit node |
|
|
| shinku-ryuu | 100.98.93.15 | Desktop workstation (Windows) |
|
|
| moon | 100.64.0.6 | Debian x86_64, GL-MT3000 subnet (`192.168.12.223`) |
|
|
| headscale-test | 100.64.0.1 | Headscale test node |
|
|
|
|
### Notes
|
|
- **moon** was migrated from public Tailscale (`dvish92@`) to Headscale on 2026-03-14. It is on the `192.168.12.0/24` subnet behind the GL-MT3000 router. `accept_routes=true` is enabled so it can reach `192.168.0.0/24` (home LAN) via Calypso's subnet advertisement.
|
|
- **guava** has `accept_routes=false` to prevent Calypso's `192.168.0.0/24` route from overriding its own LAN replies. See `docs/troubleshooting/guava-smb-incident-2026-03-14.md`.
|
|
- **shinku-ryuu** also has `accept_routes=false` for the same reason.
|
|
|
|
---
|
|
|
|
**Last Updated:** March 2026
|
|
**Note:** The Feb 2026 alerts (homelab-node and vmi2076105-node offline) were resolved. Both nodes are now online. |