154 lines
6.2 KiB
Markdown
154 lines
6.2 KiB
Markdown
# Grafana
|
|
|
|
**Monitoring Service**
|
|
|
|
## Service Overview
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| **Service Name** | grafana |
|
|
| **Host** | homelab-vm (192.168.0.210) |
|
|
| **Port** | 3300 |
|
|
| **URL** | `https://gf.vish.gg` (Authentik SSO) |
|
|
| **Category** | Monitoring |
|
|
| **Docker Image** | `grafana/grafana-oss:12.4.0` |
|
|
| **Compose File** | `hosts/vms/homelab-vm/monitoring.yaml` |
|
|
| **Stack** | `monitoring-stack` (Portainer stack ID 687, endpoint 443399) |
|
|
| **Deployment** | GitOps via Portainer |
|
|
|
|
## Purpose
|
|
|
|
Grafana is the metrics visualization and dashboarding layer for the homelab monitoring stack. It connects to Prometheus as its datasource and provides dashboards for infrastructure health, NAS metrics, and node-level detail.
|
|
|
|
## Access
|
|
|
|
| Method | URL |
|
|
|--------|-----|
|
|
| **External (SSO)** | `https://gf.vish.gg` |
|
|
| **Internal** | `http://192.168.0.210:3300` |
|
|
| **Local (on VM)** | `http://localhost:3300` |
|
|
|
|
Authentication is via **Authentik SSO** (`sso.vish.gg`). The local `admin` account is also available for API/CLI use.
|
|
|
|
## Dashboards
|
|
|
|
| Dashboard | UID | Source |
|
|
|-----------|-----|--------|
|
|
| Node Details - Full Metrics *(default home)* | `node-details-v2` | DB (imported) |
|
|
| Infrastructure Overview - All Devices | `infrastructure-overview-v2` | Provisioned (monitoring.yaml) |
|
|
| Synology NAS Monitoring | `synology-dashboard-v2` | Provisioned (monitoring.yaml) |
|
|
| Node Exporter Full | `rYdddlPWk` | DB (imported from grafana.com) |
|
|
|
|
> **Note**: `node-details-v2` and `Node Exporter Full` exist only in the `grafana-data` volume (DB). If the volume is deleted, they must be re-imported. The provisioned dashboards (Infrastructure Overview, Synology NAS) are embedded in `monitoring.yaml` and survive volume deletion.
|
|
|
|
The default home dashboard (`node-details-v2`) is set via the Grafana org preferences API and persists in the DB across container restarts.
|
|
|
|
## Configuration
|
|
|
|
### Key Environment Variables
|
|
|
|
| Variable | Value | Description |
|
|
|----------|-------|-------------|
|
|
| `GF_SECURITY_ADMIN_USER` | `admin` | Local admin username |
|
|
| `GF_SECURITY_ADMIN_PASSWORD` | `admin2024` | Local admin password (first-run only; does not override DB after initial setup) |
|
|
| `GF_FEATURE_TOGGLES_DISABLE` | `kubernetesDashboards` | Disables Grafana 12 unified storage feature toggle (prevents log spam, restores stable behavior) |
|
|
| `GF_SERVER_ROOT_URL` | `https://gf.vish.gg` | Public URL for redirect/SSO |
|
|
| `GF_AUTH_GENERIC_OAUTH_ENABLED` | `true` | Authentik SSO enabled |
|
|
|
|
### Ports
|
|
|
|
| Host Port | Container Port | Purpose |
|
|
|-----------|----------------|---------|
|
|
| 3300 | 3000 | Web interface |
|
|
|
|
### Volumes
|
|
|
|
| Volume | Container Path | Purpose |
|
|
|--------|----------------|---------|
|
|
| `monitoring-stack_grafana-data` | `/var/lib/grafana` | Persistent data (DB, plugins, sessions) |
|
|
|
|
### Provisioned Configs (Docker configs, not bind mounts)
|
|
|
|
| Config | Target | Purpose |
|
|
|--------|--------|---------|
|
|
| `grafana_datasources` | `/etc/grafana/provisioning/datasources/datasources.yaml` | Prometheus datasource |
|
|
| `grafana_dashboards_config` | `/etc/grafana/provisioning/dashboards/dashboards.yaml` | Dashboard provider config |
|
|
| `dashboard_infrastructure` | `/etc/grafana/provisioning/dashboards/json/infrastructure-overview.json` | Infrastructure Overview dashboard |
|
|
| `dashboard_synology` | `/etc/grafana/provisioning/dashboards/json/synology-monitoring.json` | Synology NAS dashboard |
|
|
|
|
## Authentik SSO
|
|
|
|
Grafana OAuth2 is configured to use Authentik at `sso.vish.gg`. Role mapping:
|
|
|
|
| Authentik Group | Grafana Role |
|
|
|-----------------|-------------|
|
|
| `Grafana Admins` | Admin |
|
|
| `Grafana Editors` | Editor |
|
|
| *(everyone else)* | Viewer |
|
|
|
|
See `docs/services/individual/grafana-oauth.md` for setup details.
|
|
|
|
## Useful Commands
|
|
|
|
```bash
|
|
# Check container status
|
|
docker ps --filter name=grafana
|
|
|
|
# View logs
|
|
docker logs grafana -f
|
|
|
|
# Reset admin password (if locked out)
|
|
docker exec grafana grafana cli --homepath /usr/share/grafana admin reset-admin-password <newpassword>
|
|
|
|
# Set org home dashboard via API
|
|
curl -X PUT http://admin:<password>@localhost:3300/api/org/preferences \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"REDACTED_APP_PASSWORD": "node-details-v2"}'
|
|
|
|
# Check current home dashboard
|
|
curl -s http://admin:<password>@localhost:3300/api/org/preferences
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Admin password not working after redeploy
|
|
`GF_SECURITY_ADMIN_PASSWORD` only applies on the very first run (empty DB). Subsequent redeployments do not reset it. Use the CLI reset:
|
|
```bash
|
|
docker exec grafana grafana cli --homepath /usr/share/grafana admin reset-admin-password <newpassword>
|
|
```
|
|
|
|
### Home dashboard reverts to Grafana welcome page
|
|
The home dashboard is stored in the `preferences` table in `grafana.db`. It survives container restarts as long as the `grafana-data` volume is not deleted. If lost, re-set it via:
|
|
```bash
|
|
curl -X PUT http://admin:<password>@localhost:3300/api/org/preferences \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"REDACTED_APP_PASSWORD": "node-details-v2"}'
|
|
```
|
|
|
|
### "No last resource version found" log spam
|
|
This is caused by the `kubernetesDashboards` feature toggle being on by default in Grafana 12. It is disabled via `GF_FEATURE_TOGGLES_DISABLE=kubernetesDashboards` in `monitoring.yaml`.
|
|
|
|
### Dashboards missing after volume wipe
|
|
Re-import `Node Details - Full Metrics` and `Node Exporter Full` from grafana.com (IDs: search grafana.com/grafana/dashboards). The provisioned dashboards (Infrastructure Overview, Synology NAS) will auto-restore from `monitoring.yaml` configs.
|
|
|
|
## Related Services
|
|
|
|
- **Prometheus** — metrics datasource (`http://prometheus:9090`)
|
|
- **Node Exporter** — host metrics (port 9100)
|
|
- **SNMP Exporter** — Synology NAS metrics (port 9116)
|
|
- **Authentik** — SSO provider (`sso.vish.gg`)
|
|
- **Nginx Proxy Manager** — reverse proxy for `gf.vish.gg`
|
|
|
|
## Related Documentation
|
|
|
|
- `docs/admin/monitoring-setup.md` — monitoring stack quick reference
|
|
- `docs/admin/monitoring.md` — full monitoring & observability guide
|
|
- `docs/services/individual/grafana-oauth.md` — Authentik SSO setup
|
|
- `docs/infrastructure/monitoring/README.md` — monitoring stack architecture
|
|
- `hosts/vms/homelab-vm/monitoring.yaml` — compose file (source of truth)
|
|
|
|
---
|
|
|
|
**Last Updated**: 2026-03-08
|
|
**Configuration Source**: `hosts/vms/homelab-vm/monitoring.yaml`
|