# Homelab Automation Scripts

> **Dashboard UI**: These scripts are visualized and controlled through the [Homelab Dashboard](../docs/services/individual/dashboard.md) at `http://homelab.tail.vish.gg:3100`. The dashboard API (`dashboard/api/`) reuses `scripts/lib/` modules and reads automation databases/logs to display real-time status. Quick actions in the dashboard can pause/resume email organizers and trigger backups.

All scripts use the local Ollama LLM (qwen3:32b) on Olares at `http://192.168.0.145:31434`.

## Shared Library (`lib/`)

Reusable modules extracted from existing scripts:

| Module | Purpose |
|--------|---------|
| `lib/ollama.py` | Ollama client with retry, backoff, think-tag stripping |
| `lib/notify.py` | ntfy push notifications + Proton Bridge SMTP email |
| `lib/portainer.py` | Portainer API client (all 5 endpoints) |
| `lib/prometheus.py` | PromQL instant and range queries |
| `lib/gitea.py` | Gitea API client (commits, releases) |

## Automated Scripts (Cron)

### Email Organizers
| Script | Account | Schedule | Description |
|--------|---------|----------|-------------|
| `gmail-organizer/gmail_organizer.py` | lzbellina92@gmail.com | Every 30min | LLM email classification + Gmail labels |
| `gmail-organizer-dvish/gmail_organizer.py` | your-email@example.com | Every 30min | LLM email classification + Gmail labels |
| `proton-organizer/proton_organizer.py` | admin@thevish.io | Every 30min | LLM email classification via Proton Bridge |

Control all 3 with: `scripts/gmail-organizer-ctl.sh [start|stop|status]`

### Monitoring
| Script | Schedule | Description |
|--------|----------|-------------|
| `backup-validator.py` | Daily 4am | Validates `/tmp/gmail-backup-daily.log`, LLM summarizes, emails report |
| `disk-predictor.py` | Sunday 9am | Prometheus disk trends, linear regression, predicts when volumes fill |
| `stack-restart.py` | Every 5min | Detects unhealthy containers, LLM assesses restart safety, auto-restarts if safe |

### Data Mining
| Script | Schedule | Description |
|--------|----------|-------------|
| `config-drift.py` | Daily 7am | Compares git compose files vs running containers via Portainer |
| `receipt-tracker.py` | Daily 10am | Extracts vendor/amount from receipt emails → `data/expenses.csv` |
| `subscription-auditor.py` | 1st of month 9am | Audits email subscriptions, flags dormant ones |

### Developer
| Script | Schedule | Description |
|--------|----------|-------------|
| `changelog-generator.py` | Monday 8am | Summarizes git commits into `docs/CHANGELOG.md` |
| `email-digest.py` | Daily 8am | Aggregates all 3 organizer stats, emails digest to admin@thevish.io |
| `pr-reviewer.py` | On PR (Gitea CI) | AI code review posted as PR comment |

## Interactive Tools (CLI)

```bash
# Generate SSH commands from plain English
python3 scripts/ssh-planner.py "restart all media services on atlantis"
python3 scripts/ssh-planner.py "check disk space on NAS boxes" --execute

# Generate Ansible playbooks from descriptions
python3 scripts/ansible-generator.py "update all hosts and reboot if kernel changed"
python3 scripts/ansible-generator.py "install tailscale on seattle" --name tailscale-seattle

# Generate troubleshooting runbooks
python3 scripts/runbook-generator.py --service jellyfin-olares
python3 scripts/runbook-generator.py --all          # all services (burst: ~30 LLM calls)
python3 scripts/runbook-generator.py --all --force   # regenerate even if unchanged
```

## Cron Setup

Add these to crontab (`crontab -e`). Email organizer crons are already installed.

```bash
# Monitoring
*/5 * * * * cd /home/homelab/organized/repos/homelab && python3 scripts/stack-restart.py >> /tmp/stack-restart.log 2>&1
0 4 * * * cd /home/homelab/organized/repos/homelab && python3 scripts/backup-validator.py >> /tmp/backup-validator.log 2>&1
0 9 * * 0 cd /home/homelab/organized/repos/homelab && python3 scripts/disk-predictor.py >> /tmp/disk-predictor.log 2>&1

# Data mining
0 7 * * * cd /home/homelab/organized/repos/homelab && python3 scripts/config-drift.py >> /tmp/config-drift.log 2>&1
0 10 * * * cd /home/homelab/organized/repos/homelab && python3 scripts/receipt-tracker.py >> /tmp/receipt-tracker.log 2>&1
0 9 1 * * cd /home/homelab/organized/repos/homelab && python3 scripts/subscription-auditor.py >> /tmp/subscription-auditor.log 2>&1

# Developer
0 8 * * 1 cd /home/homelab/organized/repos/homelab && python3 scripts/changelog-generator.py >> /tmp/changelog-generator.log 2>&1
```

## Stack Restart Safety

`stack-restart.py` has multiple safety layers:
- **Blacklist**: portainer, headscale, adguard, postgres, mariadb, mongodb, redis are never auto-restarted
- **LLM gate**: Ollama must analyze logs and say "SAFE" before any restart
- **No LLM = no restart**: if Ollama is down, it only sends alerts
- **Rate limit**: max 2 restarts per container per hour
- **Grace period**: container must be unhealthy for 5+ minutes before action
- **Dry-run**: test with `python3 scripts/stack-restart.py --dry-run`

## Ollama Usage

| Category | Calls/Day | Notes |
|----------|-----------|-------|
| Email organizers (3) | 0-15 | Sender cache reduces calls over time |
| Monitoring (3) | 1-3 | stack-restart only fires on unhealthy |
| Data mining (3) | 2-6 | receipt-tracker scales with purchases |
| Developer (2) | 0-1 | changelog weekly, PR reviews on demand |
| Interactive (3) | 0-5 | only when you use them |
| **Total** | **~5-15** | **Negligible for RTX 5090** |

## Graceful Degradation

All automated scripts check `ollama_available()` before making LLM calls:
- **Monitoring scripts**: send alerts with raw data (no LLM summary)
- **Data mining scripts**: skip run, retry next schedule
- **Interactive tools**: print error and exit

## MCP Server Integration

The homelab MCP server (`homelab-mcp/server.py`) includes an `ollama_query` tool that lets Claude Code query the local LLM directly. This saves API tokens for homelab-specific questions.

## Logs

All cron scripts log to `/tmp/`:
```bash
tail -f /tmp/stack-restart.log
tail -f /tmp/backup-validator.log
tail -f /tmp/disk-predictor.log
tail -f /tmp/config-drift.log
tail -f /tmp/receipt-tracker.log
tail -f /tmp/subscription-auditor.log
tail -f /tmp/changelog-generator.log
tail -f /tmp/email-digest.log
```