132 lines
6.1 KiB
Markdown
132 lines
6.1 KiB
Markdown
# Homelab Automation Scripts
|
|
|
|
> **Dashboard UI**: These scripts are visualized and controlled through the [Homelab Dashboard](../docs/services/individual/dashboard.md) at `http://homelab.tail.vish.gg:3100`. The dashboard API (`dashboard/api/`) reuses `scripts/lib/` modules and reads automation databases/logs to display real-time status. Quick actions in the dashboard can pause/resume email organizers and trigger backups.
|
|
|
|
All scripts use the local Ollama LLM (qwen3:32b) on Olares at `http://192.168.0.145:31434`.
|
|
|
|
## Shared Library (`lib/`)
|
|
|
|
Reusable modules extracted from existing scripts:
|
|
|
|
| Module | Purpose |
|
|
|--------|---------|
|
|
| `lib/ollama.py` | Ollama client with retry, backoff, think-tag stripping |
|
|
| `lib/notify.py` | ntfy push notifications + Proton Bridge SMTP email |
|
|
| `lib/portainer.py` | Portainer API client (all 5 endpoints) |
|
|
| `lib/prometheus.py` | PromQL instant and range queries |
|
|
| `lib/gitea.py` | Gitea API client (commits, releases) |
|
|
|
|
## Automated Scripts (Cron)
|
|
|
|
### Email Organizers
|
|
| Script | Account | Schedule | Description |
|
|
|--------|---------|----------|-------------|
|
|
| `gmail-organizer/gmail_organizer.py` | lzbellina92@gmail.com | Every 30min | LLM email classification + Gmail labels |
|
|
| `gmail-organizer-dvish/gmail_organizer.py` | your-email@example.com | Every 30min | LLM email classification + Gmail labels |
|
|
| `proton-organizer/proton_organizer.py` | admin@thevish.io | Every 30min | LLM email classification via Proton Bridge |
|
|
|
|
Control all 3 with: `scripts/gmail-organizer-ctl.sh [start|stop|status]`
|
|
|
|
### Monitoring
|
|
| Script | Schedule | Description |
|
|
|--------|----------|-------------|
|
|
| `backup-validator.py` | Daily 4am | Validates `/tmp/gmail-backup-daily.log`, LLM summarizes, emails report |
|
|
| `disk-predictor.py` | Sunday 9am | Prometheus disk trends, linear regression, predicts when volumes fill |
|
|
| `stack-restart.py` | Every 5min | Detects unhealthy containers, LLM assesses restart safety, auto-restarts if safe |
|
|
|
|
### Data Mining
|
|
| Script | Schedule | Description |
|
|
|--------|----------|-------------|
|
|
| `config-drift.py` | Daily 7am | Compares git compose files vs running containers via Portainer |
|
|
| `receipt-tracker.py` | Daily 10am | Extracts vendor/amount from receipt emails → `data/expenses.csv` |
|
|
| `subscription-auditor.py` | 1st of month 9am | Audits email subscriptions, flags dormant ones |
|
|
|
|
### Developer
|
|
| Script | Schedule | Description |
|
|
|--------|----------|-------------|
|
|
| `changelog-generator.py` | Monday 8am | Summarizes git commits into `docs/CHANGELOG.md` |
|
|
| `email-digest.py` | Daily 8am | Aggregates all 3 organizer stats, emails digest to admin@thevish.io |
|
|
| `pr-reviewer.py` | On PR (Gitea CI) | AI code review posted as PR comment |
|
|
|
|
## Interactive Tools (CLI)
|
|
|
|
```bash
|
|
# Generate SSH commands from plain English
|
|
python3 scripts/ssh-planner.py "restart all media services on atlantis"
|
|
python3 scripts/ssh-planner.py "check disk space on NAS boxes" --execute
|
|
|
|
# Generate Ansible playbooks from descriptions
|
|
python3 scripts/ansible-generator.py "update all hosts and reboot if kernel changed"
|
|
python3 scripts/ansible-generator.py "install tailscale on seattle" --name tailscale-seattle
|
|
|
|
# Generate troubleshooting runbooks
|
|
python3 scripts/runbook-generator.py --service jellyfin-olares
|
|
python3 scripts/runbook-generator.py --all # all services (burst: ~30 LLM calls)
|
|
python3 scripts/runbook-generator.py --all --force # regenerate even if unchanged
|
|
```
|
|
|
|
## Cron Setup
|
|
|
|
Add these to crontab (`crontab -e`). Email organizer crons are already installed.
|
|
|
|
```bash
|
|
# Monitoring
|
|
*/5 * * * * cd /home/homelab/organized/repos/homelab && python3 scripts/stack-restart.py >> /tmp/stack-restart.log 2>&1
|
|
0 4 * * * cd /home/homelab/organized/repos/homelab && python3 scripts/backup-validator.py >> /tmp/backup-validator.log 2>&1
|
|
0 9 * * 0 cd /home/homelab/organized/repos/homelab && python3 scripts/disk-predictor.py >> /tmp/disk-predictor.log 2>&1
|
|
|
|
# Data mining
|
|
0 7 * * * cd /home/homelab/organized/repos/homelab && python3 scripts/config-drift.py >> /tmp/config-drift.log 2>&1
|
|
0 10 * * * cd /home/homelab/organized/repos/homelab && python3 scripts/receipt-tracker.py >> /tmp/receipt-tracker.log 2>&1
|
|
0 9 1 * * cd /home/homelab/organized/repos/homelab && python3 scripts/subscription-auditor.py >> /tmp/subscription-auditor.log 2>&1
|
|
|
|
# Developer
|
|
0 8 * * 1 cd /home/homelab/organized/repos/homelab && python3 scripts/changelog-generator.py >> /tmp/changelog-generator.log 2>&1
|
|
```
|
|
|
|
## Stack Restart Safety
|
|
|
|
`stack-restart.py` has multiple safety layers:
|
|
- **Blacklist**: portainer, headscale, adguard, postgres, mariadb, mongodb, redis are never auto-restarted
|
|
- **LLM gate**: Ollama must analyze logs and say "SAFE" before any restart
|
|
- **No LLM = no restart**: if Ollama is down, it only sends alerts
|
|
- **Rate limit**: max 2 restarts per container per hour
|
|
- **Grace period**: container must be unhealthy for 5+ minutes before action
|
|
- **Dry-run**: test with `python3 scripts/stack-restart.py --dry-run`
|
|
|
|
## Ollama Usage
|
|
|
|
| Category | Calls/Day | Notes |
|
|
|----------|-----------|-------|
|
|
| Email organizers (3) | 0-15 | Sender cache reduces calls over time |
|
|
| Monitoring (3) | 1-3 | stack-restart only fires on unhealthy |
|
|
| Data mining (3) | 2-6 | receipt-tracker scales with purchases |
|
|
| Developer (2) | 0-1 | changelog weekly, PR reviews on demand |
|
|
| Interactive (3) | 0-5 | only when you use them |
|
|
| **Total** | **~5-15** | **Negligible for RTX 5090** |
|
|
|
|
## Graceful Degradation
|
|
|
|
All automated scripts check `ollama_available()` before making LLM calls:
|
|
- **Monitoring scripts**: send alerts with raw data (no LLM summary)
|
|
- **Data mining scripts**: skip run, retry next schedule
|
|
- **Interactive tools**: print error and exit
|
|
|
|
## MCP Server Integration
|
|
|
|
The homelab MCP server (`homelab-mcp/server.py`) includes an `ollama_query` tool that lets Claude Code query the local LLM directly. This saves API tokens for homelab-specific questions.
|
|
|
|
## Logs
|
|
|
|
All cron scripts log to `/tmp/`:
|
|
```bash
|
|
tail -f /tmp/stack-restart.log
|
|
tail -f /tmp/backup-validator.log
|
|
tail -f /tmp/disk-predictor.log
|
|
tail -f /tmp/config-drift.log
|
|
tail -f /tmp/receipt-tracker.log
|
|
tail -f /tmp/subscription-auditor.log
|
|
tail -f /tmp/changelog-generator.log
|
|
tail -f /tmp/email-digest.log
|
|
```
|