121 lines
5.7 KiB
Markdown
121 lines
5.7 KiB
Markdown
# AI Integrations
|
|
|
|
**Last updated:** 2026-03-20
|
|
|
|
Overview of all AI/LLM integrations across the homelab. The primary GPU inference backend is **Olares** (RTX 5090 Max-Q, 24GB VRAM) running Qwen3-Coder via Ollama.
|
|
|
|
---
|
|
|
|
## Primary AI Backend — Olares
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| **Host** | Olares (`192.168.0.145`) |
|
|
| **GPU** | RTX 5090 Max-Q (24GB VRAM) |
|
|
| **Active model** | `qwen3-coder:latest` (30.5B MoE, Q4_K_M) |
|
|
| **Ollama endpoint** | `https://a5be22681.vishinator.olares.com` |
|
|
| **OpenAI-compat endpoint** | `https://a5be22681.vishinator.olares.com/v1` |
|
|
| **Native Ollama API** | `https://a5be22681.vishinator.olares.com/api/...` |
|
|
|
|
> Port 11434 is not directly exposed — all access goes through the Olares reverse proxy at the above URL.
|
|
|
|
### Check active models
|
|
```bash
|
|
curl -s https://a5be22681.vishinator.olares.com/api/tags | python3 -m json.tool
|
|
curl -s https://a5be22681.vishinator.olares.com/api/ps # currently loaded in VRAM
|
|
```
|
|
|
|
### Switch models
|
|
See `docs/services/individual/olares.md` for scaling operations.
|
|
|
|
---
|
|
|
|
## Services Using Olares AI
|
|
|
|
| Service | Host | Feature | Config |
|
|
|---------|------|---------|--------|
|
|
| **AnythingLLM** | Atlantis | RAG document assistant | `LLM_PROVIDER=generic-openai`, `GENERIC_OPEN_AI_BASE_PATH=https://a5be22681.vishinator.olares.com/v1`, model=`qwen3-coder:latest` |
|
|
| **Perplexica** | homelab-vm | AI-powered search engine | `OLLAMA_BASE_URL=https://a5be22681.vishinator.olares.com`, model set via UI |
|
|
| **Reactive Resume v5** | Calypso | AI resume writing assistance | `OPENAI_BASE_URL=https://a5be22681.vishinator.olares.com/v1`, model=`qwen3-coder:latest` |
|
|
| **OpenCode (homelab-vm)** | homelab-vm | Coding agent | `~/.config/opencode/opencode.json` → Olares Ollama, model=`qwen3-coder:latest` |
|
|
| **OpenCode (moon)** | moon | Coding agent | `/home/moon/.config/opencode/opencode.json` → Olares Ollama, model=`qwen3-coder:latest` (was: vLLM `qwen3-30b` — migrated 2026-03-20) |
|
|
|
|
### Perplexica config persistence
|
|
Perplexica stores its provider config in a Docker volume at `/home/perplexica/data/config.json`. The `OLLAMA_BASE_URL` env var sets the default but the UI/DB config takes precedence. The current config is set to `olares-ollama` provider with `qwen3-coder:latest`.
|
|
|
|
To reset if the config gets corrupted:
|
|
```bash
|
|
docker exec perplexica cat /home/perplexica/data/config.json
|
|
# Edit and update as needed, then restart
|
|
docker restart perplexica
|
|
```
|
|
|
|
---
|
|
|
|
## Services Using Other AI Backends
|
|
|
|
| Service | Host | Backend | Notes |
|
|
|---------|------|---------|-------|
|
|
| **OpenHands** | homelab-vm | Anthropic Claude Sonnet 4 (cloud) | `LLM_MODEL=anthropic/claude-sonnet-4-20250514` — kept on Claude as it's significantly better for agentic coding than local models |
|
|
| **Paperless-AI** | Calypso | LM Studio on Shinku (`100.98.93.15:1234`) via Tailscale | Auto-tags/classifies Paperless documents. Model: `llama-3.2-3b-instruct`. Could be switched to Olares for better quality. |
|
|
| **Hoarder** | homelab-vm | OpenAI cloud API (`sk-proj-...`) | AI bookmark tagging/summarization. Could be switched to Olares to save cost. |
|
|
| **Home Assistant Voice** | Concord NUC | Local Whisper `tiny-int8` + Piper TTS | Voice command pipeline — fully local, no GPU needed |
|
|
| **Ollama + Open WebUI** | Atlantis | ROCm GPU (`phi3:mini`, `gemma:2b`) | Separate Ollama instance for Atlantis-local use |
|
|
| **LlamaGPT** | Atlantis | llama.cpp (`Nous-Hermes-Llama-2-7B`) | Legacy — likely unused |
|
|
| **Reactive Resume (bundled)** | Calypso | Bundled Ollama `Resume-OLLAMA-V5` (`llama3.2:3b`) | Still running but app is now pointed at Olares |
|
|
| **Ollama + vLLM** | Seattle VPS | CPU-only (`llama3.2:3b`, `Qwen2.5-1.5B`) | CPU inference, used previously by Perplexica |
|
|
| **OpenHands (MSI laptop)** | Edge device | LM Studio (`devstral-small-2507`) | Ad-hoc run config, not a managed stack |
|
|
|
|
---
|
|
|
|
## Candidates to Migrate to Olares
|
|
|
|
| Service | Effort | Benefit |
|
|
|---------|--------|---------|
|
|
| **Paperless-AI** | Low — change `CUSTOM_BASE_URL` in compose | Better model (30B vs 3B) for document classification |
|
|
| **Hoarder** | Low — add `OPENAI_BASE_URL` env var | Eliminates cloud API cost |
|
|
|
|
---
|
|
|
|
## Olares Endpoint Reference
|
|
|
|
| Protocol | URL | Use for |
|
|
|----------|-----|---------|
|
|
| OpenAI-compat (Ollama) | `https://a5be22681.vishinator.olares.com/v1` | Services expecting OpenAI API format — **primary endpoint** |
|
|
| Native Ollama | `https://a5be22681.vishinator.olares.com` | Services with native Ollama support |
|
|
| Models list | `https://a5be22681.vishinator.olares.com/api/tags` | Check available models |
|
|
| Active models | `https://a5be22681.vishinator.olares.com/api/ps` | Check VRAM usage |
|
|
| vLLM (legacy) | `https://04521407.vishinator.olares.com/v1` | vLLM inference — available but not currently used |
|
|
|
|
> **Note:** Only one large model should be loaded at a time (24GB VRAM limit). If inference is slow or failing, check `api/ps` — another model may be occupying VRAM.
|
|
|
|
### OpenCode per-host config
|
|
|
|
OpenCode config lives at `~/.config/opencode/opencode.json` on each machine. All instances use Olares Ollama:
|
|
|
|
```json
|
|
{
|
|
"$schema": "https://opencode.ai/config.json",
|
|
"provider": {
|
|
"olares": {
|
|
"npm": "@ai-sdk/openai-compatible",
|
|
"name": "Olares Ollama (Qwen3-Coder)",
|
|
"options": {
|
|
"baseURL": "https://a5be22681.vishinator.olares.com/v1"
|
|
},
|
|
"models": {
|
|
"qwen3-coder:latest": {
|
|
"name": "Qwen3 Coder 30.5B Q4_K_M",
|
|
"limit": { "context": 40000, "output": 8192 }
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"model": "olares/qwen3-coder:latest"
|
|
}
|
|
```
|
|
|
|
Config locations:
|
|
- **homelab-vm**: `/home/homelab/.config/opencode/opencode.json`
|
|
- **moon**: `/home/moon/.config/opencode/opencode.json` (migrated from vLLM 2026-03-20)
|