857 lines
38 KiB
Markdown
857 lines
38 KiB
Markdown
# 🏗️ Service Architecture
|
|
|
|
## Overview
|
|
|
|
This document shows how the ~240 Docker services (plus Olares K8s) interact, their dependencies, and the data flows between them. For a list of recently added services (HolyClaude, Dashboard, email digest infra, tdarr nodes), see [CHANGELOG.md](../CHANGELOG.md) and [INDEX.md](../INDEX.md).
|
|
|
|
---
|
|
|
|
## 🎬 Media Stack Architecture (Mermaid)
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph Internet["☁️ Internet Sources"]
|
|
USENET["Usenet<br/>Providers"]
|
|
TORRENT["Torrent<br/>Trackers"]
|
|
INDEXERS["Indexers<br/>(NZB/Torrent)"]
|
|
end
|
|
|
|
subgraph Acquisition["📥 Content Acquisition (Atlantis)"]
|
|
PROWLARR["Prowlarr<br/>Indexer Manager"]
|
|
SONARR["Sonarr<br/>TV Shows"]
|
|
RADARR["Radarr<br/>Movies"]
|
|
LIDARR["Lidarr<br/>Music"]
|
|
READARR["Readarr<br/>Books"]
|
|
WHISPARR["Whisparr<br/>Adult"]
|
|
BAZARR["Bazarr<br/>Subtitles"]
|
|
|
|
SAB["SABnzbd<br/>Usenet Client"]
|
|
DELUGE["Deluge<br/>Torrent Client<br/>(via Gluetun VPN)"]
|
|
end
|
|
|
|
subgraph Storage["💾 Storage (Atlantis NAS)"]
|
|
MEDIA_TV["/volume1/media/tv"]
|
|
MEDIA_MOV["/volume1/media/movies"]
|
|
MEDIA_MUS["/volume1/media/music"]
|
|
MEDIA_BOOK["/volume1/media/books"]
|
|
end
|
|
|
|
subgraph Streaming["📺 Media Streaming"]
|
|
PLEX["Plex<br/>Media Server"]
|
|
JELLYFIN["Jellyfin<br/>Media Server"]
|
|
TAUTULLI["Tautulli<br/>Plex Analytics"]
|
|
end
|
|
|
|
subgraph Clients["📱 Client Devices"]
|
|
TV["Smart TVs"]
|
|
PHONE["Phones/Tablets"]
|
|
WEB["Web Browsers"]
|
|
APPS["Desktop Apps"]
|
|
end
|
|
|
|
%% Acquisition flow
|
|
INDEXERS --> PROWLARR
|
|
PROWLARR --> SONARR & RADARR & LIDARR & READARR & WHISPARR
|
|
|
|
SONARR --> SAB & DELUGE
|
|
RADARR --> SAB & DELUGE
|
|
LIDARR --> SAB & DELUGE
|
|
READARR --> SAB & DELUGE
|
|
WHISPARR --> SAB & DELUGE
|
|
|
|
USENET --> SAB
|
|
TORRENT --> DELUGE
|
|
|
|
%% Storage flow
|
|
SAB --> MEDIA_TV & MEDIA_MOV & MEDIA_MUS & MEDIA_BOOK
|
|
DELUGE --> MEDIA_TV & MEDIA_MOV & MEDIA_MUS & MEDIA_BOOK
|
|
|
|
BAZARR --> MEDIA_TV & MEDIA_MOV
|
|
|
|
%% Streaming flow
|
|
MEDIA_TV & MEDIA_MOV --> PLEX & JELLYFIN
|
|
|
|
PLEX --> TAUTULLI
|
|
|
|
%% Client access
|
|
PLEX & JELLYFIN --> TV & PHONE & WEB & APPS
|
|
|
|
classDef acquisition fill:#e74c3c,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef storage fill:#3498db,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef streaming fill:#2ecc71,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef client fill:#9b59b6,stroke:#333,stroke-width:2px,color:#fff
|
|
|
|
class PROWLARR,SONARR,RADARR,LIDARR,READARR,WHISPARR,BAZARR,SAB,DELUGE acquisition
|
|
class MEDIA_TV,MEDIA_MOV,MEDIA_MUS,MEDIA_BOOK storage
|
|
class PLEX,JELLYFIN,TAUTULLI streaming
|
|
class TV,PHONE,WEB,APPS client
|
|
```
|
|
|
|
---
|
|
|
|
## 📊 Monitoring Stack Architecture
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph Targets["🎯 Monitored Targets"]
|
|
subgraph Synology["Synology NAS"]
|
|
ATL_SNMP["Atlantis<br/>SNMP"]
|
|
CAL_SNMP["Calypso<br/>SNMP"]
|
|
SET_SNMP["Setillo<br/>SNMP"]
|
|
end
|
|
|
|
subgraph Hosts["Linux Hosts"]
|
|
NODE1["Homelab VM<br/>node_exporter"]
|
|
NODE2["Guava<br/>node_exporter"]
|
|
NODE3["Anubis<br/>node_exporter"]
|
|
end
|
|
|
|
subgraph Containers["Containers"]
|
|
CADV["cAdvisor<br/>Container Metrics"]
|
|
end
|
|
|
|
subgraph Network["Network"]
|
|
BLACK["Blackbox Exporter<br/>HTTP/ICMP Probes"]
|
|
end
|
|
end
|
|
|
|
subgraph Collection["📥 Metric Collection (Homelab VM)"]
|
|
PROM["Prometheus<br/>Time Series DB"]
|
|
SNMP_EXP["SNMP Exporter"]
|
|
end
|
|
|
|
subgraph Visualization["📈 Visualization"]
|
|
GRAFANA["Grafana<br/>Dashboards"]
|
|
end
|
|
|
|
subgraph Alerting["🚨 Alerting"]
|
|
ALERTMGR["Alertmanager"]
|
|
NTFY["ntfy<br/>Push Notifications"]
|
|
UPTIME["Uptime Kuma<br/>Status Page"]
|
|
end
|
|
|
|
%% Collection
|
|
ATL_SNMP & CAL_SNMP & SET_SNMP --> SNMP_EXP
|
|
SNMP_EXP --> PROM
|
|
NODE1 & NODE2 & NODE3 --> PROM
|
|
CADV --> PROM
|
|
BLACK --> PROM
|
|
|
|
%% Visualization
|
|
PROM --> GRAFANA
|
|
PROM --> ALERTMGR
|
|
ALERTMGR --> NTFY
|
|
|
|
%% Uptime Kuma separate
|
|
BLACK -.-> UPTIME
|
|
|
|
classDef target fill:#e67e22,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef collection fill:#3498db,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef viz fill:#2ecc71,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef alert fill:#e74c3c,stroke:#333,stroke-width:2px,color:#fff
|
|
|
|
class ATL_SNMP,CAL_SNMP,SET_SNMP,NODE1,NODE2,NODE3,CADV,BLACK target
|
|
class PROM,SNMP_EXP collection
|
|
class GRAFANA viz
|
|
class ALERTMGR,NTFY,UPTIME alert
|
|
```
|
|
|
|
---
|
|
|
|
## 🔐 Authentication & Security Stack
|
|
|
|
### Complete Authentication Architecture
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph External["🌐 External Access"]
|
|
USERS["👤 Users"]
|
|
CLOUDFLARE["☁️ Cloudflare<br/>DNS/WAF/DDoS"]
|
|
end
|
|
|
|
subgraph Gateway["🚪 Gateway Layer (matrix-ubuntu)"]
|
|
NPM["🔀 Nginx Proxy Manager<br/>matrix-ubuntu :81/:443<br/>Reverse Proxy + SSL"]
|
|
CFT["🚇 Cloudflare Tunnel<br/>Zero Trust Access"]
|
|
end
|
|
|
|
subgraph AuthLayer["🔐 Authentication Layer (Calypso)"]
|
|
AUTH_SRV["🔐 Authentik Server<br/>:9000"]
|
|
AUTH_PROXY["🛡️ Authentik Outpost<br/>:9444<br/>Forward Auth Proxy"]
|
|
AUTH_WRK["⚙️ Authentik Worker"]
|
|
AUTH_DB["🐘 PostgreSQL"]
|
|
AUTH_RED["🔴 Redis"]
|
|
end
|
|
|
|
subgraph VPN["🔒 VPN Layer"]
|
|
WIREGUARD["🔒 Wireguard<br/>Atlantis :51820"]
|
|
TAILSCALE["🔷 Tailscale<br/>100.x.x.x"]
|
|
HEADSCALE["🌐 Headscale<br/>Calypso :8080"]
|
|
end
|
|
|
|
subgraph DNS["🌐 DNS & Ad Blocking"]
|
|
ADGUARD1["🛡️ AdGuard<br/>Calypso :53"]
|
|
ADGUARD2["🛡️ AdGuard<br/>Atlantis :53"]
|
|
ADGUARD3["🛡️ AdGuard<br/>NUC :53"]
|
|
end
|
|
|
|
subgraph SecVault["🔑 Secrets Management"]
|
|
VAULT["🔑 Vaultwarden<br/>vault.vish.gg"]
|
|
end
|
|
|
|
subgraph ProtectedServices["🛡️ Protected Services"]
|
|
GRAFANA["📊 Grafana"]
|
|
PAPERLESS["📄 Paperless"]
|
|
IMMICH["📸 Immich"]
|
|
ACTUAL["💰 Actual Budget"]
|
|
GITEA["🔧 Gitea"]
|
|
NETBOX["🔌 NetBox"]
|
|
HOMARR["🏠 Homarr"]
|
|
RXRESUME["📝 Reactive Resume"]
|
|
HEADPLANE["🌐 Headplane"]
|
|
end
|
|
|
|
subgraph PublicServices["🌍 Public/Self-Auth Services"]
|
|
PLEX["📺 Plex"]
|
|
SEAFILE["☁️ Seafile"]
|
|
OST["🚀 OpenSpeedTest"]
|
|
NTFY["📣 ntfy"]
|
|
end
|
|
|
|
%% External flow
|
|
USERS --> CLOUDFLARE
|
|
CLOUDFLARE --> NPM
|
|
CLOUDFLARE --> CFT
|
|
USERS --> TAILSCALE
|
|
|
|
%% NPM to Auth
|
|
NPM -->|"Forward Auth<br/>Header Check"| AUTH_PROXY
|
|
AUTH_PROXY -->|"Validate Session"| AUTH_SRV
|
|
|
|
%% Auth internal
|
|
AUTH_SRV --> AUTH_DB
|
|
AUTH_SRV --> AUTH_RED
|
|
AUTH_WRK --> AUTH_DB
|
|
AUTH_WRK --> AUTH_RED
|
|
|
|
%% Protected services via NPM + Auth
|
|
NPM -->|"✓ Authenticated"| ProtectedServices
|
|
|
|
%% Public services direct
|
|
NPM --> PublicServices
|
|
|
|
%% VPN access
|
|
TAILSCALE --> HEADSCALE
|
|
WIREGUARD --> ProtectedServices
|
|
TAILSCALE --> ProtectedServices
|
|
|
|
%% DNS
|
|
ADGUARD1 -.-> ProtectedServices
|
|
ADGUARD2 -.-> PublicServices
|
|
|
|
classDef external fill:#e74c3c,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef gateway fill:#f39c12,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef auth fill:#9b59b6,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef dns fill:#1abc9c,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef protected fill:#3498db,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef public fill:#27ae60,stroke:#333,stroke-width:2px,color:#fff
|
|
|
|
class USERS,CLOUDFLARE external
|
|
class NPM,CFT gateway
|
|
class AUTH_SRV,AUTH_PROXY,AUTH_WRK,AUTH_DB,AUTH_RED,VAULT auth
|
|
class ADGUARD1,ADGUARD2,ADGUARD3 dns
|
|
class GRAFANA,PAPERLESS,IMMICH,ACTUAL,GITEA,NETBOX,HOMARR,RXRESUME,HEADPLANE protected
|
|
class PLEX,SEAFILE,OST,NTFY public
|
|
```
|
|
|
|
---
|
|
|
|
### Authentik SSO Flow (Detailed)
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
autonumber
|
|
participant U as 👤 User
|
|
participant CF as ☁️ Cloudflare
|
|
participant NPM as 🔀 NPM (matrix-ubuntu)
|
|
participant OUT as 🛡️ Outpost (Calypso)
|
|
participant AUTH as 🔐 Authentik (Calypso)
|
|
participant APP as 📱 Application
|
|
|
|
U->>CF: Request app.vish.gg
|
|
CF->>NPM: Forward (HTTPS)
|
|
NPM->>OUT: Forward Auth Request<br/>(/outpost.goauthentik.io/auth/nginx)
|
|
|
|
alt No Valid Session
|
|
OUT->>AUTH: Check Session
|
|
AUTH-->>OUT: No Session
|
|
OUT-->>NPM: 401 Unauthorized
|
|
NPM-->>U: Redirect to sso.vish.gg/flows/default-authentication/
|
|
U->>AUTH: Login Page
|
|
U->>AUTH: Submit Credentials + 2FA
|
|
AUTH->>AUTH: Validate
|
|
AUTH-->>U: Set Cookie + Redirect to app
|
|
U->>NPM: Retry with Session Cookie
|
|
NPM->>OUT: Forward Auth (with cookie)
|
|
end
|
|
|
|
OUT->>AUTH: Validate Session
|
|
AUTH-->>OUT: Valid ✓
|
|
OUT-->>NPM: 200 OK + Headers<br/>(X-authentik-username, X-authentik-email)
|
|
NPM->>APP: Proxy Request (with auth headers)
|
|
APP-->>U: Response
|
|
```
|
|
|
|
---
|
|
|
|
### NPM Proxy Host Configuration
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph NPM["🔀 Nginx Proxy Manager (matrix-ubuntu :81)"]
|
|
subgraph ProxyHosts["Proxy Hosts"]
|
|
PH1["sso.vish.gg → Calypso:9000"]
|
|
PH2["git.vish.gg → Calypso:3052"]
|
|
PH3["gf.vish.gg → homelab-vm:3300"]
|
|
PH4["nb.vish.gg → homelab-vm:8443"]
|
|
PH5["ntfy.vish.gg → homelab-vm:8081"]
|
|
PH6["dash.vish.gg → Atlantis:7575"]
|
|
PH7["paperless.vish.gg → Calypso:8777"]
|
|
PH8["rx.vish.gg → Calypso:4550"]
|
|
PH9["actual.vish.gg → Calypso:8304"]
|
|
PH10["kuma.vish.gg → RPi5:3001"]
|
|
end
|
|
|
|
subgraph SSL["SSL Certificates"]
|
|
WILD["*.vish.gg<br/>Cloudflare DNS Challenge"]
|
|
end
|
|
|
|
subgraph AccessControl["Access Control"]
|
|
AUTH_LOC["Authentik Forward Auth<br/>Location: /outpost.goauthentik.io"]
|
|
end
|
|
end
|
|
|
|
subgraph Services["Backend Services"]
|
|
direction LR
|
|
S1["Authentik"]
|
|
S2["Gitea"]
|
|
S3["Grafana"]
|
|
S4["NetBox"]
|
|
S5["ntfy"]
|
|
S6["Homarr"]
|
|
S7["Paperless"]
|
|
S8["Reactive Resume"]
|
|
S9["Actual"]
|
|
S10["Uptime Kuma"]
|
|
end
|
|
|
|
PH1 --> S1
|
|
PH2 --> S2
|
|
PH3 --> S3
|
|
PH4 --> S4
|
|
PH5 --> S5
|
|
PH6 --> S6
|
|
PH7 --> S7
|
|
PH8 --> S8
|
|
PH9 --> S9
|
|
PH10 --> S10
|
|
```
|
|
|
|
---
|
|
|
|
### Services Protected by Authentik
|
|
|
|
| Domain | Service | Host | Auth Type | Notes |
|
|
|--------|---------|------|-----------|-------|
|
|
| `sso.vish.gg` | Authentik | Calypso | - | Identity Provider |
|
|
| `git.vish.gg` | Gitea | Calypso | OAuth2/OIDC | Source Control |
|
|
| `gf.vish.gg` | Grafana | Homelab VM | OAuth2/OIDC | Monitoring |
|
|
| `nb.vish.gg` | NetBox | Homelab VM | OAuth2/OIDC | DCIM/IPAM |
|
|
| `dash.vish.gg` | Homarr | Atlantis | OAuth2/OIDC | Dashboard |
|
|
| `rx.vish.gg` | Reactive Resume | Calypso | OAuth2/OIDC | Resume Builder |
|
|
| `immich` | Immich | Calypso | OAuth2/OIDC | Photos |
|
|
| `headscale.vish.gg/admin` | Headplane | Calypso | OAuth2/OIDC | VPN Admin |
|
|
| `paperless.vish.gg` | Paperless-NGX | Calypso | Forward Auth | Documents |
|
|
| `actual.vish.gg` | Actual Budget | Calypso | Forward Auth | Finance |
|
|
|
|
### Services NOT Protected (Public/Self-Auth)
|
|
|
|
| Domain | Service | Host | Reason |
|
|
|--------|---------|------|--------|
|
|
| `plex.vish.gg` | Plex | Atlantis | Has Plex Auth |
|
|
| `sf.vish.gg` | Seafile | Calypso | Has built-in auth + share links |
|
|
| `ntfy.vish.gg` | ntfy | Homelab | Has built-in auth + public topics |
|
|
| `ost.vish.gg` | OpenSpeedTest | Calypso | Public utility |
|
|
|
|
---
|
|
|
|
### Authentik Forward Auth Setup (NPM)
|
|
|
|
To protect a service with Authentik Forward Auth in NPM:
|
|
|
|
1. **Create Provider in Authentik**:
|
|
- Type: Proxy Provider
|
|
- External Host: `https://app.vish.gg`
|
|
- Mode: Forward auth (single application)
|
|
|
|
2. **Create Application in Authentik**:
|
|
- Link to the provider
|
|
- Set policies for access control
|
|
|
|
3. **Create Outpost in Authentik**:
|
|
- Type: Proxy
|
|
- Include the application
|
|
|
|
4. **Configure NPM Proxy Host**:
|
|
```nginx
|
|
# Custom Nginx Configuration (Advanced tab)
|
|
|
|
# Authentik Forward Auth
|
|
location /outpost.goauthentik.io {
|
|
proxy_pass http://calypso.vish.local:9444/outpost.goauthentik.io;
|
|
proxy_set_header Host $host;
|
|
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
|
|
add_header Set-Cookie $auth_cookie;
|
|
auth_request_set $auth_cookie $upstream_http_set_cookie;
|
|
proxy_pass_request_body off;
|
|
proxy_set_header Content-Length "";
|
|
}
|
|
|
|
location / {
|
|
auth_request /outpost.goauthentik.io/auth/nginx;
|
|
error_page 401 = @goauthentik_proxy_signin;
|
|
auth_request_set $auth_cookie $upstream_http_set_cookie;
|
|
add_header Set-Cookie $auth_cookie;
|
|
|
|
# Forward auth headers to application
|
|
auth_request_set $authentik_username $upstream_http_x_authentik_username;
|
|
auth_request_set $authentik_email $upstream_http_x_authentik_email;
|
|
proxy_set_header X-authentik-username $authentik_username;
|
|
proxy_set_header X-authentik-email $authentik_email;
|
|
|
|
proxy_pass http://backend;
|
|
}
|
|
|
|
location @goauthentik_proxy_signin {
|
|
internal;
|
|
add_header Set-Cookie $auth_cookie;
|
|
return 302 /outpost.goauthentik.io/start?rd=$request_uri;
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 📝 ASCII Service Distribution by Host
|
|
|
|
```
|
|
╔══════════════════════════════════════════════════════════════════════════════════════════╗
|
|
║ SERVICE DISTRIBUTION BY HOST ║
|
|
╚══════════════════════════════════════════════════════════════════════════════════════════╝
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🏛️ ATLANTIS (51 Containers) - Media & Communication Hub │
|
|
│ ═══════════════════════════════════════════════════════════════════════════════════════│
|
|
│ │
|
|
│ 📺 Media 🔐 Security 🛠️ Infrastructure │
|
|
│ ───────────── ───────────── ───────────────── │
|
|
│ • Plex • Vaultwarden • Portainer │
|
|
│ • Jellyfin • Wireguard • DokuWiki │
|
|
│ • Immich • Dozzle │
|
|
│ • Tautulli • Watchtower │
|
|
│ • Homarr (dash) • IT-Tools │
|
|
│ • AdGuard Home (backup DNS) │
|
|
│ │
|
|
│ 💬 Communication 📝 Productivity 🎮 Other │
|
|
│ ───────────── ───────────── ───────────── │
|
|
│ • Matrix Synapse • Documenso • Stirling PDF │
|
|
│ • Mastodon • Joplin Server • YouTube DL │
|
|
│ • Mattermost │
|
|
└─────────────────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🏢 CALYPSO (54 Containers) - Auth, Proxy, Arr Suite & Development │
|
|
│ ═══════════════════════════════════════════════════════════════════════════════════════│
|
|
│ │
|
|
│ 🔐 Auth 📥 Arr Suite 💻 Development 📦 Infrastructure │
|
|
│ ───────────── ───────────── ───────────── ───────────── │
|
|
│ • Authentik • Sonarr • Gitea • Headscale │
|
|
│ • Authentik Outpost • Radarr • Reactive Resume • AdGuard Home │
|
|
│ • Lidarr • Seafile • Portainer Agent │
|
|
│ • Readarr • Wireguard │
|
|
│ 💰 Finance • Prowlarr 📝 Productivity │
|
|
│ ───────────── • SABnzbd ───────────── │
|
|
│ • Actual Budget • Deluge (Gluetun) • Paperless-NGX │
|
|
│ • Bazarr • Rustdesk │
|
|
│ • Whisparr │
|
|
└─────────────────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 💻 HOMELAB VM (30 Containers) - Monitoring, Tools & Privacy │
|
|
│ ═══════════════════════════════════════════════════════════════════════════════════════│
|
|
│ │
|
|
│ 📊 Monitoring 🔔 Notifications 🔌 DCIM 🔧 Utilities │
|
|
│ ───────────── ───────────── ───────────── ───────────── │
|
|
│ • Grafana • ntfy • NetBox • Archivebox │
|
|
│ • Prometheus • Signal-API • Hoarder │
|
|
│ • Alertmanager 🔒 Privacy • Perplexica │
|
|
│ • SNMP Exporter 🤖 AI/Dev ───────────── • OpenHands │
|
|
│ • node_exporter ───────────── • Redlib │
|
|
│ • OpenHands • Binternet │
|
|
│ • Perplexica • ProxiTok │
|
|
└─────────────────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🌐 CONCORD NUC (19 Containers) - Home Automation & Edge │
|
|
│ ═══════════════════════════════════════════════════════════════════════════════════════│
|
|
│ │
|
|
│ 🏠 Home Automation 📺 Media 🎵 Music 🔧 Network │
|
|
│ ───────────── ───────────── ───────────── ───────────── │
|
|
│ • Home Assistant • Plex • Your-Spotify • AdGuard Home │
|
|
│ • Matter Server • Invidious • Wireguard │
|
|
│ • Whisper (STT) │
|
|
│ • Piper (TTS) │
|
|
│ • OpenWakeWord │
|
|
└─────────────────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🍓 RPi 5 (3 Containers) - Monitoring │
|
|
│ ═══════════════════════════════════════════════════════════════════════════════════════│
|
|
│ │
|
|
│ 📊 Monitoring │
|
|
│ ───────────── │
|
|
│ • Uptime Kuma │
|
|
│ • Glances │
|
|
│ • Portainer Agent │
|
|
└─────────────────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🤖 OLARES - K8s Node (Core Ultra 9 275HX, RTX 5090, 96GB) │
|
|
│ ═══════════════════════════════════════════════════════════════════════════════════════│
|
|
│ │
|
|
│ 🧠 AI/ML (Kubernetes, not Docker) │
|
|
│ ───────────────────────────────── │
|
|
│ • Ollama (LLM serving) │
|
|
│ • vLLM (high-throughput inference) │
|
|
│ • OpenClaw (robotics foundation model) │
|
|
└─────────────────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🌵 SETILLO (4 Services) - Tucson Remote │
|
|
│ ═══════════════════════════════════════════════════════════════════════════════════════│
|
|
│ │
|
|
│ 📊 Monitoring 🌐 DNS │
|
|
│ ───────────── ───────────── │
|
|
│ • Prometheus • AdGuard Home │
|
|
│ • SNMP Exporter • Syncthing │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
╔════════════════════════════════════════════════════════════════════════════════════════╗
|
|
║ SERVICE COUNT SUMMARY ║
|
|
║ ═════════════════════ ║
|
|
║ Atlantis: 59 containers │ Calypso: 61 containers ║
|
|
║ Homelab VM: 38 containers │ Concord NUC: 19 containers ║
|
|
║ RPi 5: 6 containers │ matrix-ubuntu: 12+ containers (NPM, Matrix) ║
|
|
║ Olares: K8s (~60 pods, not Portainer) ║
|
|
║ ──────────────────────────────────────────────────────────────────────────────────────║
|
|
║ TOTAL: ~195 containers across 5 Portainer endpoints + matrix-ubuntu + Olares ║
|
|
╚════════════════════════════════════════════════════════════════════════════════════════╝
|
|
```
|
|
|
|
---
|
|
|
|
## 🔗 Related Diagrams
|
|
- [Network Topology](network-topology.md) - How hosts connect
|
|
- [Storage Topology](storage-topology.md) - Where data lives
|
|
- [Tailscale Mesh](tailscale-mesh.md) - Cross-location access
|
|
|
|
---
|
|
|
|
## 💬 Communication Stack Architecture
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph Internet["☁️ Internet / Federation"]
|
|
FEDI["Fediverse<br/>(ActivityPub)"]
|
|
MATRIX_FED["Matrix<br/>Federation"]
|
|
WEBRTC["WebRTC<br/>Voice/Video"]
|
|
end
|
|
|
|
subgraph Cloudflare["🛡️ Cloudflare"]
|
|
CF_PROXY["Cloudflare<br/>Proxy/WAF"]
|
|
CF_TUNNEL["Cloudflare<br/>Tunnel"]
|
|
end
|
|
|
|
subgraph MatrixUbuntuVM["🐧 Matrix-Ubuntu VM (Atlantis)"]
|
|
subgraph Mastodon["🐘 Mastodon Stack"]
|
|
MASTO_WEB["Mastodon Web<br/>:3000"]
|
|
MASTO_STREAM["Mastodon Streaming<br/>:4000"]
|
|
MASTO_SIDEKIQ["Sidekiq<br/>Background Jobs"]
|
|
end
|
|
|
|
subgraph Matrix["🔐 Matrix Stack"]
|
|
SYNAPSE["Synapse<br/>:8008 / :8018"]
|
|
ELEMENT["Element Web<br/>Client"]
|
|
COTURN["Coturn<br/>TURN Server<br/>:3478"]
|
|
end
|
|
|
|
subgraph Mattermost["💬 Mattermost"]
|
|
MM_APP["Mattermost<br/>:8065"]
|
|
end
|
|
|
|
subgraph SharedDB["🗄️ Shared Services"]
|
|
POSTGRES["PostgreSQL<br/>:5432"]
|
|
REDIS["Redis<br/>:6379"]
|
|
end
|
|
|
|
NPM_VM["NPM<br/>Reverse Proxy<br/>(host nginx disabled)"]
|
|
end
|
|
|
|
subgraph Atlantis["🏛️ Atlantis NAS"]
|
|
subgraph JitsiStack["📹 Jitsi Meet"]
|
|
JITSI_WEB["Jitsi Web"]
|
|
JITSI_JVB["Jitsi Video Bridge"]
|
|
JITSI_PROSODY["Prosody XMPP"]
|
|
end
|
|
|
|
subgraph Vaultwarden["🔑 Vaultwarden"]
|
|
VW["Vaultwarden<br/>Password Manager"]
|
|
end
|
|
|
|
subgraph Joplin["📝 Joplin"]
|
|
JOPLIN_SRV["Joplin Server"]
|
|
end
|
|
end
|
|
|
|
subgraph Clients["📱 Clients"]
|
|
BROWSER["Web Browsers"]
|
|
MOBILE["Mobile Apps"]
|
|
DESKTOP["Desktop Apps"]
|
|
end
|
|
|
|
%% External connections
|
|
FEDI <--> CF_PROXY
|
|
MATRIX_FED <--> CF_PROXY
|
|
WEBRTC <--> COTURN
|
|
|
|
%% Cloudflare to services
|
|
CF_PROXY --> NPM_VM
|
|
CF_TUNNEL --> NPM_VM
|
|
|
|
%% NPM routing (host nginx disabled, NPM handles all)
|
|
NPM_VM --> MASTO_WEB & MASTO_STREAM
|
|
NPM_VM --> SYNAPSE & ELEMENT
|
|
NPM_VM --> MM_APP
|
|
|
|
%% Database connections
|
|
MASTO_WEB & MASTO_SIDEKIQ --> POSTGRES & REDIS
|
|
SYNAPSE --> POSTGRES
|
|
MM_APP --> POSTGRES
|
|
|
|
%% Client access
|
|
BROWSER & MOBILE & DESKTOP --> CF_PROXY
|
|
BROWSER & MOBILE & DESKTOP --> JITSI_WEB
|
|
BROWSER & MOBILE & DESKTOP --> VW
|
|
BROWSER & MOBILE & DESKTOP --> JOPLIN_SRV
|
|
|
|
classDef mastodon fill:#6364FF,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef matrix fill:#0DBD8B,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef mattermost fill:#0058CC,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef infra fill:#e67e22,stroke:#333,stroke-width:2px,color:#fff
|
|
|
|
class MASTO_WEB,MASTO_STREAM,MASTO_SIDEKIQ mastodon
|
|
class SYNAPSE,ELEMENT,COTURN matrix
|
|
class MM_APP mattermost
|
|
class POSTGRES,REDIS,NPM_VM infra
|
|
```
|
|
|
|
### Communication Services Summary
|
|
|
|
| Service | Domain | Protocol | Purpose |
|
|
|---------|--------|----------|---------|
|
|
| **Mastodon** | mastodon.vish.gg | ActivityPub | Fediverse microblogging |
|
|
| **Matrix (Primary)** | mx.vish.gg | Matrix | Federated chat |
|
|
| **Matrix (Legacy)** | matrix.thevish.io | Matrix | Legacy homeserver |
|
|
| **Mattermost** | mm.crista.love | Proprietary | Team collaboration |
|
|
| **Jitsi Meet** | meet.vish.gg | WebRTC | Video conferencing |
|
|
| **Joplin** | joplin.vish.gg | Joplin Sync | Note synchronization |
|
|
| **Vaultwarden** | vault.vish.gg | Bitwarden | Password management |
|
|
|
|
### Deployment Scripts
|
|
|
|
| Script | Location | Description |
|
|
|--------|----------|-------------|
|
|
| Mastodon Install | [mastodon-production/](../mastodon-production/) | Bare metal & Docker deployment |
|
|
| Matrix Install | [matrix-element/](../matrix-element/) | Synapse + Element + TURN |
|
|
| Mattermost Install | [mattermost-production/](../mattermost-production/) | Docker deployment |
|
|
| VM Config | [matrix-ubuntu-vm/](../matrix-ubuntu-vm/) | Complete VM configuration |
|
|
|
|
---
|
|
|
|
## 🔄 CI/CD Pipeline Architecture
|
|
|
|
### Git Repository Mirroring
|
|
|
|
The homelab repository uses Gitea Actions for automated CI/CD, including sanitized public mirroring.
|
|
|
|
```mermaid
|
|
graph LR
|
|
subgraph Development["💻 Development"]
|
|
DEV["Developer<br/>Pushes Code"]
|
|
end
|
|
|
|
subgraph Gitea["🔧 Gitea (Calypso)"]
|
|
PRIVATE["🔒 Private Repo<br/>homelab"]
|
|
PUBLIC["🌐 Public Repo<br/>homelab-optimized"]
|
|
RUNNER["🏃 Gitea Runners<br/>(homelab, calypso, pi5)"]
|
|
end
|
|
|
|
subgraph Workflow["⚙️ CI/CD Workflow"]
|
|
CHECKOUT["📥 Checkout Code"]
|
|
SANITIZE["🧹 Sanitize<br/>Remove Secrets"]
|
|
PUSH["📤 Force Push<br/>Fresh History"]
|
|
end
|
|
|
|
subgraph Deployment["🚀 Deployment"]
|
|
ANSIBLE["📋 Ansible<br/>Multi-host"]
|
|
PORTAINER["🐳 Portainer<br/>5 Endpoints"]
|
|
end
|
|
|
|
DEV -->|"git push"| PRIVATE
|
|
PRIVATE -->|"Triggers"| RUNNER
|
|
RUNNER --> CHECKOUT
|
|
CHECKOUT --> SANITIZE
|
|
SANITIZE --> PUSH
|
|
PUSH --> PUBLIC
|
|
|
|
PRIVATE --> ANSIBLE
|
|
ANSIBLE --> PORTAINER
|
|
```
|
|
|
|
### Sanitization Process
|
|
|
|
The sanitization script removes sensitive data before public mirroring:
|
|
|
|
| Removed | Pattern | Example |
|
|
|---------|---------|---------|
|
|
| Passwords | `password:`, `PASS=` | `password: "REDACTED_PASSWORD" |
|
|
| API Keys | `api_key:`, `API_KEY=` | `api_key: REDACTED_API_KEY` |
|
|
| Tokens | `token:`, `TOKEN=` | `token: REDACTED_TOKEN` |
|
|
| Secrets | `secret:`, `SECRET=` | `secret: REDACTED_SECRET` |
|
|
| Private Keys | `-----BEGIN.*KEY-----` | File removed |
|
|
| SSH Keys | `id_rsa`, `id_ed25519` | File removed |
|
|
| Personal Emails | `*@gmail.com`, `*@*.com` | `REDACTED_EMAIL@example.com` |
|
|
| JWT Secrets | `JWT_SECRET=` | `JWT_SECRET=REDACTED` |
|
|
|
|
### Gitea Runner Setup
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph Calypso["🌊 Calypso (DS723+)"]
|
|
GITEA["🔧 Gitea Server<br/>:3052"]
|
|
RUNNER_CAL["🏃 Runner (calypso)"]
|
|
end
|
|
|
|
subgraph HomelabVM["💻 Homelab VM"]
|
|
RUNNER_HLB["🏃 Runner (homelab)"]
|
|
end
|
|
|
|
subgraph Pi5["🍓 RPi 5"]
|
|
RUNNER_PI["🏃 Runner (pi5)"]
|
|
end
|
|
|
|
GITEA -->|"Workflow Dispatch"| RUNNER_CAL
|
|
GITEA -->|"Workflow Dispatch"| RUNNER_HLB
|
|
GITEA -->|"Workflow Dispatch"| RUNNER_PI
|
|
```
|
|
|
|
**Runner Configuration:**
|
|
- Runner binary: `act_runner` v0.2.6, systemd service (not Docker container)
|
|
- Labels: `ubuntu-latest`, `linux`, `python` (all 3 runners)
|
|
- Runners: homelab (VM), calypso, pi5
|
|
- Trigger: Push to main branch
|
|
|
|
### Ansible Automation
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph Control["📋 Ansible Control"]
|
|
SITE["site.yml<br/>Master Playbook"]
|
|
INV["inventory.yml<br/>13 Hosts"]
|
|
ROLES["Roles<br/>docker_stack, directory_setup"]
|
|
end
|
|
|
|
subgraph Hosts["🖥️ Target Hosts"]
|
|
SYN["Synology<br/>Atlantis, Calypso, Setillo"]
|
|
VMS["VMs<br/>Homelab, matrix-ubuntu"]
|
|
PHYS["Physical<br/>Guava, NUC, Shinku-Ryuu"]
|
|
EDGE["Edge<br/>RPi5, Jellyfish"]
|
|
CLOUD["Cloud<br/>Seattle VPS"]
|
|
end
|
|
|
|
SITE --> INV
|
|
INV --> SYN
|
|
INV --> VMS
|
|
INV --> PHYS
|
|
INV --> EDGE
|
|
INV --> CLOUD
|
|
```
|
|
|
|
**Ansible Commands:**
|
|
```bash
|
|
# Deploy everything
|
|
ansible-playbook site.yml
|
|
|
|
# Deploy to specific host
|
|
ansible-playbook site.yml --limit atlantis
|
|
|
|
# Deploy by category
|
|
ansible-playbook site.yml --tags synology
|
|
|
|
# Check status
|
|
ansible-playbook playbooks/common/status.yml
|
|
```
|
|
|
|
---
|
|
|
|
## 🧠 AI/ML Stack Architecture
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph Olares["🤖 Olares K8s Node (Core Ultra 9 275HX, RTX 5090, 96GB)"]
|
|
OLLAMA["🦙 Ollama<br/>LLM Serving<br/>Local Models"]
|
|
VLLM["⚡ vLLM<br/>High-Throughput<br/>Inference Engine"]
|
|
OPENCLAW["🤖 OpenClaw<br/>Robotics Foundation<br/>Model"]
|
|
end
|
|
|
|
subgraph Clients["📱 AI Consumers"]
|
|
ANYTHINGLLM["💬 AnythingLLM<br/>RAG Chat"]
|
|
OPENWEBUI["🌐 Open WebUI"]
|
|
API_CLIENTS["🔧 API Clients"]
|
|
end
|
|
|
|
OLLAMA -->|"OpenAI-compatible API"| Clients
|
|
VLLM -->|"OpenAI-compatible API"| Clients
|
|
|
|
classDef ai fill:#8e44ad,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef client fill:#2980b9,stroke:#333,stroke-width:2px,color:#fff
|
|
|
|
class OLLAMA,VLLM,OPENCLAW ai
|
|
class ANYTHINGLLM,OPENWEBUI,API_CLIENTS client
|
|
```
|
|
|
|
### AI/ML Services Summary
|
|
|
|
| Service | Host | Type | Purpose |
|
|
|---------|------|------|---------|
|
|
| **Ollama** | Olares (K8s) | LLM Server | Local model serving (Llama, Mistral, etc.) |
|
|
| **vLLM** | Olares (K8s) | Inference Engine | High-throughput batched inference |
|
|
| **OpenClaw** | Olares (K8s) | Foundation Model | Robotics/manipulation research |
|
|
| **AnythingLLM** | Homelab VM | RAG Client | Document Q&A with local LLMs |
|
|
|
|
---
|
|
|
|
## 🔗 Related Diagrams
|
|
- [Network Topology](network-topology.md) - How hosts connect
|
|
- [Storage Topology](storage-topology.md) - Where data lives
|
|
- [Tailscale Mesh](tailscale-mesh.md) - Cross-location access
|