4.7 KiB
4.7 KiB
Homelab Monitoring Architecture
This document explains the different monitoring setups in the homelab and their purposes.
🏗️ Architecture Overview
The homelab has three distinct monitoring deployments serving different purposes:
1. Production GitOps Monitoring (Primary)
- Location:
hosts/vms/homelab-vm/monitoring.yaml - Deployment: Portainer GitOps on homelab-vm
- Purpose: Production monitoring for all homelab infrastructure
- Access: https://gf.vish.gg (with Authentik SSO)
- Status: ✅ ACTIVE - This is the canonical monitoring stack
Features:
- Monitors all homelab devices (Synology NAS, nodes, VMs)
- Authentik OAuth2 SSO integration
- Embedded dashboard configs in Docker Compose
- Auto-provisioned datasources and dashboards
- SNMP monitoring for Synology devices
2. Fixed Development Stack (New)
- Location:
docker/monitoring/ - Deployment: Standalone Docker Compose
- Purpose: Development/testing with fixed dashboard issues
- Access: http://localhost:3300 (admin/admin)
- Status: 🔧 DEVELOPMENT - For testing and dashboard fixes
Features:
- All dashboard datasource UIDs fixed
- Template variables working correctly
- Instance filters properly configured
- Verification scripts included
- Backup/restore functionality
3. Atlantis Legacy Setup (Deprecated)
- Location:
hosts/synology/atlantis/grafana_prometheus/ - Deployment: Synology Docker on Atlantis
- Purpose: Legacy monitoring setup
- Status: 📦 ARCHIVED - Kept for reference
🔄 GitOps Workflow
Production Deployment (homelab-vm)
# GitOps automatically deploys from:
hosts/vms/homelab-vm/monitoring.yaml
# Portainer Stack Details:
# - Stack ID: 476
# - Endpoint: 443399
# - Auto-updates from git repository
Development Testing (docker/monitoring)
# Manual deployment for testing:
cd docker/monitoring
docker-compose up -d
# Verify dashboards:
./verify-dashboard-sections.sh
📊 Dashboard Status
| Dashboard | Production (GitOps) | Development (Fixed) | Status |
|---|---|---|---|
| Infrastructure Overview | ✅ Working | ✅ Fixed | Both functional |
| Synology NAS Monitoring | ⚠️ Needs UID fix | ✅ Fixed | Dev has fixes |
| Node Exporter Full | ⚠️ Needs UID fix | ✅ Fixed | Dev has fixes |
| Node Details | ⚠️ Needs UID fix | ✅ Fixed | Dev has fixes |
🔧 Applying Fixes to Production
To apply the dashboard fixes to the production GitOps deployment:
- Extract fixed dashboards from
docker/monitoring/grafana/dashboards/ - Update the embedded configs in
hosts/vms/homelab-vm/monitoring.yaml - Test locally using the development stack
- Commit changes - GitOps will auto-deploy
Example: Updating Synology Dashboard in GitOps
# 1. Extract the fixed dashboard JSON
cat docker/monitoring/grafana/dashboards/synology-nas-monitoring.json
# 2. Update the embedded config in monitoring.yaml
# Replace the dashboard_synology config content with the fixed JSON
# 3. Commit and push - GitOps handles deployment
git add hosts/vms/homelab-vm/monitoring.yaml
git commit -m "Fix Synology dashboard datasource UID in GitOps"
git push
🚀 Deployment Commands
Production (GitOps - Automatic)
# No manual deployment needed
# Portainer GitOps auto-deploys from git repository
# Access: https://gf.vish.gg
Development (Manual)
cd docker/monitoring
docker-compose up -d
# Access: http://localhost:3300
Legacy (Manual - Not Recommended)
cd hosts/synology/atlantis/grafana_prometheus
# Deploy via Synology Docker UI
📋 Maintenance
Updating Production Dashboards
- Test fixes in
docker/monitoring/first - Update embedded configs in
hosts/vms/homelab-vm/monitoring.yaml - Commit changes for GitOps auto-deployment
Backup Strategy
- Production: Automated via GitOps repository
- Development: Use
backup.shandrestore.shscripts - Legacy: Manual Synology backup
🔍 Troubleshooting
Dashboard "No Data" Issues
- Check datasource UID matches Prometheus instance
- Verify template variables have correct queries
- Ensure instance filters are not empty
- Use development stack to test fixes first
GitOps Deployment Issues
- Check Portainer stack logs
- Verify git repository connectivity
- Ensure Docker configs are valid YAML
- Test locally with development stack