homelab-optimized/docs/services/individual/olares.md

# Olares

**Kubernetes Self-Hosting Platform**

## Service Overview

| Property | Value |
|----------|-------|
| **Host** | olares (192.168.0.145) |
| **OS** | Ubuntu 24.04.3 LTS |
| **Platform** | Olares (Kubernetes/K3s with Calico CNI) |
| **Hardware** | Intel Core Ultra 9 275HX, 96GB DDR5, RTX 5090 Max-Q, 2TB NVMe |
| **SSH** | `ssh olares` (key auth, user: olares) |

## Purpose

Olares is a Kubernetes-based self-hosting platform running on a high-end mini PC. It provides a managed app store for deploying containerized services with built-in auth (Authelia), networking (Envoy sidecars), and GPU scheduling (HAMI).

Primary use case: **local LLM inference** via vLLM and Ollama, exposed as OpenAI-compatible API endpoints for coding agents (OpenCode, OpenClaw).

## LLM Services

Models are deployed via the Olares app store and served as OpenAI-compatible APIs. Each model gets a unique subdomain under `*.vishinator.olares.com`.

### Available Models

| Model | Backend | Namespace | Endpoint | Context | Notes |
|-------|---------|-----------|----------|---------|-------|
| Qwen3-Coder 30B | Ollama | `ollamaserver-shared` | `https://a5be22681.vishinator.olares.com/v1` | 65k tokens | MoE (3.3B active), coding-focused, currently active |
| Qwen3 30B A3B (4-bit) | vLLM | `vllmqwen330ba3bv2server-shared` | `https://04521407.vishinator.olares.com/v1` | ~40k tokens | MoE, fast inference, limited tool calling |
| Qwen3 30B A3B (4-bit) | vLLM | `vllmqwen330ba3binstruct4bitv2-vishinator` | — | ~40k tokens | Duplicate deployment (vishinator namespace) |
| Qwen3.5 27B Q4_K_M | Ollama | `ollamaqwen3527bq4kmv2server-shared` | `https://37e62186.vishinator.olares.com/v1` | 40k+ (262k native) | Dense, best for agentic coding |
| GPT-OSS 20B | vLLM | `vllmgptoss20bv2server-shared` | `https://6941bf89.vishinator.olares.com/v1` | 65k tokens | Requires auth bypass in Olares settings |
| Qwen3.5 9B | Ollama | `ollamaqwen359bv2server-shared` | — | — | Lightweight, scaled to 0 |

### GPU Memory Constraints (RTX 5090 Max-Q, 24 GB VRAM)

- Only run **one model at a time** to avoid VRAM exhaustion
- vLLM `--gpu-memory-utilization 0.95` is the default
- Context limits are determined by available KV cache after model loading
- Use `nvidia-smi` or check vLLM logs for actual KV cache capacity
- Before starting a model, scale down all others (see Scaling Operations below)

### Scaling Operations

Only one model should be loaded at a time due to VRAM constraints. Use these commands to switch between models.

**Check what's running:**
```bash
ssh olares "sudo kubectl get deployments -A | grep -iE 'vllm|ollama'"
ssh olares "nvidia-smi --query-gpu=memory.used,memory.free --format=csv"
```

**Stop all LLM deployments (free GPU):**
```bash
# Qwen3-Coder (Ollama — currently active)
ssh olares "sudo kubectl scale deployment ollama -n ollamaserver-shared --replicas=0"
ssh olares "sudo kubectl scale deployment terminal -n ollamaserver-shared --replicas=0"

# Qwen3 30B A3B vLLM (shared)
ssh olares "sudo kubectl scale deployment vllm -n vllmqwen330ba3bv2server-shared --replicas=0"

# Qwen3 30B A3B vLLM (vishinator)
ssh olares "sudo kubectl scale deployment vllmqwen330ba3binstruct4bitv2 -n vllmqwen330ba3binstruct4bitv2-vishinator --replicas=0"

# Qwen3.5 27B Ollama
ssh olares "sudo kubectl scale deployment ollama -n ollamaqwen3527bq4kmv2server-shared --replicas=0"
ssh olares "sudo kubectl scale deployment api -n ollamaqwen3527bq4kmv2server-shared --replicas=0"

# GPT-OSS 20B vLLM
ssh olares "sudo kubectl scale deployment vllm -n vllmgptoss20bv2server-shared --replicas=0"
```

**Start Qwen3-Coder (Ollama):**
```bash
ssh olares "sudo kubectl scale deployment ollama -n ollamaserver-shared --replicas=1"
ssh olares "sudo kubectl scale deployment terminal -n ollamaserver-shared --replicas=1"
```

**Start Qwen3 30B A3B (vLLM):**
```bash
ssh olares "sudo kubectl scale deployment vllm -n vllmqwen330ba3bv2server-shared --replicas=1"
# Wait 2-3 minutes for vLLM startup, then check:
ssh olares "sudo kubectl logs -n vllmqwen330ba3bv2server-shared -l io.kompose.service=vllm --tail=5"
```

**Start Qwen3.5 27B (Ollama):**
```bash
ssh olares "sudo kubectl scale deployment ollama -n ollamaqwen3527bq4kmv2server-shared --replicas=1"
ssh olares "sudo kubectl scale deployment api -n ollamaqwen3527bq4kmv2server-shared --replicas=1"
```

**Unload a model from Ollama (without scaling down the pod):**
```bash
ssh olares "sudo kubectl exec -n ollamaserver-shared \$(sudo kubectl get pods -n ollamaserver-shared -l io.kompose.service=ollama -o jsonpath='{.items[0].metadata.name}') -c ollama -- ollama stop qwen3-coder:latest"
```

### vLLM max_model_len

The `max_model_len` parameter is set in the deployment command args. To check the hardware-safe maximum, look at vLLM startup logs:

```
Available KV cache memory: X.XX GiB
GPU KV cache size: XXXXX tokens
```

To change it, either:
1. Edit in the **Olares app settings UI** (persistent across redeploys)
2. Patch the deployment directly (resets on redeploy):
   ```bash
   kubectl get deployment vllm -n <namespace> -o json > /tmp/patch.json
   # Edit max-model-len in the command string
   kubectl apply -f /tmp/patch.json
   ```

## OpenClaw (Chat Agent)

OpenClaw runs as a Kubernetes app in the `clawdbot-vishinator` namespace.

### Configuration

Config file inside the pod: `/home/node/.openclaw/openclaw.json`

To read/write config:
```bash
ssh olares
sudo kubectl exec -n clawdbot-vishinator <pod> -c clawdbot -- cat /home/node/.openclaw/openclaw.json
```

### Key Settings

- **Compaction**: `mode: "safeguard"` with `maxHistoryShare: 0.5` prevents context overflow
- **contextWindow**: Must match vLLM's actual `max_model_len` (not the model's native limit)
- **Workspace data**: Lives at `/home/node/.openclaw/workspace/` inside the pod
- **Brew packages**: OpenClaw has Homebrew; install tools with `brew install <pkg>` from the agent or pod

### Troubleshooting

| Error | Cause | Fix |
|-------|-------|-----|
| `localhost:8000 connection refused` | Model provider not configured or not running | Check model endpoint URL in config, verify vLLM pod is running |
| `Context overflow` | Prompt exceeded model's context limit | Enable compaction, or `/reset` the session |
| `pairing required` (WebSocket 1008) | Device pairing data was cleared | Reload the Control UI page to re-pair |
| `does not support tools` (400) | Ollama model lacks tool calling template | Use vLLM with `--enable-auto-tool-choice` instead of Ollama |
| `max_tokens must be at least 1, got negative` | Context window too small for system prompt + tools | Increase `max_model_len` (vLLM) or `num_ctx` (Ollama) |
| `bad request` / 400 from Ollama | Request exceeds `num_ctx` | Increase `num_ctx` in Modelfile: `ollama create model -f Modelfile` |
| 302 redirect on model endpoint | Olares auth (Authelia) blocking API access | Disable auth for the endpoint in Olares app settings |
| vLLM server pod scaled to 0 | Previously stopped, client pod crashes | Scale up: `kubectl scale deployment vllm -n <namespace> --replicas=1` |

## OpenCode Configuration

OpenCode on the homelab VM and moon are configured to use these endpoints.

### Config Location

- **homelab VM**: `~/.config/opencode/opencode.json`
- **moon**: `~/.config/opencode/opencode.json` (user: moon)

### Model Switching

Change the `"model"` field in `opencode.json`:

```json
"model": "olares//models/qwen3-30b"
```

Available provider/model strings:
- `olares//models/qwen3-30b` (recommended — supports tool calling via vLLM)
- `olares-gptoss//models/gpt-oss-20b`
- `olares-qwen35/qwen3.5:27b-q4_K_M` (Ollama — does NOT support tool calling, avoid for OpenCode)

**Important**: OpenCode requires tool/function calling support. Ollama models often lack tool call templates, causing 400 errors. Use vLLM with `--enable-auto-tool-choice --tool-call-parser hermes` for reliable tool use.

### Loop Prevention

```json
"mode": {
  "build": {
    "steps": 25,
    "permission": { "doom_loop": "deny" }
  },
  "plan": {
    "steps": 15,
    "permission": { "doom_loop": "deny" }
  }
}
```

## Storage — NFS Mount from Atlantis

Olares has an NFS mount from Atlantis for persistent storage shared with the homelab:

| Property | Value |
|----------|-------|
| **Mount point** | `/mnt/atlantis_olares_storage` |
| **Source** | `192.168.0.200:/volume1/documents/olares_storage` |
| **Access** | Read/write (`all_squash`, anonuid=1026/anongid=100) |
| **Persistent** | Yes — configured in `/etc/fstab` |
| **Capacity** | 84TB pool (46TB free as of 2026-03-16) |

### fstab entry
```
192.168.0.200:/volume1/documents/olares_storage /mnt/atlantis_olares_storage nfs rw,async,hard,intr,rsize=8192,wsize=8192,timeo=14 0 0
```

### Mount/unmount manually
```bash
# Mount
sudo mount /mnt/atlantis_olares_storage

# Unmount
sudo umount /mnt/atlantis_olares_storage

# Check
df -h /mnt/atlantis_olares_storage
ls /mnt/atlantis_olares_storage
```

### Troubleshooting
- If mount fails after reboot, check Atlantis is up and NFS is running: `sudo showmount -e 192.168.0.200`
- Fail2ban on Olares may ban homelab-vm (`192.168.0.210`) — whitelist is `/etc/fail2ban/jail.d/local.conf` with `ignoreip = 127.0.0.1/8 ::1 192.168.0.0/24`
- SSH to Olares uses key auth (`ssh olares` works from homelab-vm) — key installed 2026-03-16

---

## Built-in Services

Olares runs its own infrastructure in Kubernetes:

- **Headscale + Tailscale**: Internal mesh network (separate tailnet from homelab, IP 100.64.0.1)
- **Authelia**: SSO/auth gateway for app endpoints
- **Envoy**: Sidecar proxy for all apps
- **HAMI**: GPU device scheduler for vLLM/Ollama pods
- **Prometheus**: Metrics collection

## Network

| Interface | IP | Notes |
|-----------|-----|-------|
| LAN (enp129s0) | 192.168.0.145/24 | Primary access |
| Tailscale (K8s pod) | 100.64.0.1 | Olares internal tailnet only |

Note: The host does **not** have Tailscale installed directly. The K8s Tailscale pod uses `tailscale0` and conflicts with host-level tailscale (causes network outage if both run). Access via LAN only.

## Known Issues

- **Do NOT install host-level Tailscale** — it conflicts with the K8s Tailscale pod's `tailscale0` interface and causes total network loss requiring physical reboot
- **Ollama Qwen3.5 27B lacks tool calling** — Ollama's model template doesn't support tools; use vLLM for coding agents
- **Only run one model at a time** — running multiple vLLM instances exhausts 24GB VRAM; scale unused deployments to 0
- **vLLM startup takes 2-3 minutes** — requests during startup return 502/connection refused; wait for "Application startup complete" in logs
- **Olares auth (Authelia) blocks API endpoints by default** — new model endpoints need auth bypass configured in Olares app settings

## Remote Management with k9s

k9s and kubectl are installed on the homelab VM for managing Olares pods without SSH.

### Setup

| Component | Details |
|-----------|---------|
| **kubectl** | `/usr/local/bin/kubectl` (v1.35.2) |
| **k9s** | `/usr/local/bin/k9s` (v0.50.18) |
| **kubeconfig** | `~/.kube/config` → `https://192.168.0.145:6443` |
| **Access** | Full admin (K3s default user), LAN only |

The kubeconfig was copied from `/etc/rancher/k3s/k3s.yaml` on Olares with the server address changed from `127.0.0.1` to `192.168.0.145`.

### Usage

```bash
# Launch k9s (interactive TUI)
k9s

# Filter by namespace
k9s -n ollamaserver-shared

# Quick kubectl checks
kubectl get pods -A
kubectl get deployments -A | grep -iE 'ollama|vllm'
kubectl logs -n ollamaserver-shared -l io.kompose.service=ollama --tail=20
kubectl scale deployment ollama -n ollamaserver-shared --replicas=0
```

### Limitations

- **LAN only** — Olares has no host-level Tailscale, so k9s only works from the local network
- **Metrics API not available** — `kubectl top` / k9s resource view won't work
- **Kubeconfig rotation** — if Olares is reinstalled or K3s certs rotate, re-copy the kubeconfig:
  ```bash
  ssh olares "sudo cat /etc/rancher/k3s/k3s.yaml" | sed 's|https://127.0.0.1:6443|https://192.168.0.145:6443|' > ~/.kube/config
  chmod 600 ~/.kube/config
  ```

## Maintenance

### Reboot
```bash
ssh olares 'sudo reboot'
```
Allow 3-5 minutes for K8s pods to come back up. Check with:
```bash
ssh olares 'sudo kubectl get pods -A | grep -v Running'
```

### Memory Management
With 96 GB RAM, multiple models can load into system memory but GPU VRAM is the bottleneck. Monitor with:
```bash
ssh olares 'free -h; nvidia-smi'
```