794 lines
34 KiB
Markdown
794 lines
34 KiB
Markdown
# 🏗️ Service Architecture
|
|
|
|
## Overview
|
|
|
|
This document shows how the 176+ Docker services interact, their dependencies, and the data flows between them.
|
|
|
|
---
|
|
|
|
## 🎬 Media Stack Architecture (Mermaid)
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph Internet["☁️ Internet Sources"]
|
|
USENET["Usenet<br/>Providers"]
|
|
TORRENT["Torrent<br/>Trackers"]
|
|
INDEXERS["Indexers<br/>(NZB/Torrent)"]
|
|
end
|
|
|
|
subgraph Acquisition["📥 Content Acquisition (Atlantis)"]
|
|
PROWLARR["Prowlarr<br/>Indexer Manager"]
|
|
SONARR["Sonarr<br/>TV Shows"]
|
|
RADARR["Radarr<br/>Movies"]
|
|
LIDARR["Lidarr<br/>Music"]
|
|
READARR["Readarr<br/>Books"]
|
|
BAZARR["Bazarr<br/>Subtitles"]
|
|
|
|
SAB["SABnzbd<br/>Usenet Client"]
|
|
QBIT["qBittorrent<br/>Torrent Client"]
|
|
end
|
|
|
|
subgraph Storage["💾 Storage (Atlantis NAS)"]
|
|
MEDIA_TV["/volume1/media/tv"]
|
|
MEDIA_MOV["/volume1/media/movies"]
|
|
MEDIA_MUS["/volume1/media/music"]
|
|
MEDIA_BOOK["/volume1/media/books"]
|
|
end
|
|
|
|
subgraph Streaming["📺 Media Streaming"]
|
|
PLEX["Plex<br/>Media Server"]
|
|
JELLYFIN["Jellyfin<br/>Media Server"]
|
|
NAVIDROME["Navidrome<br/>Music Server"]
|
|
TAUTULLI["Tautulli<br/>Plex Analytics"]
|
|
end
|
|
|
|
subgraph Clients["📱 Client Devices"]
|
|
TV["Smart TVs"]
|
|
PHONE["Phones/Tablets"]
|
|
WEB["Web Browsers"]
|
|
APPS["Desktop Apps"]
|
|
end
|
|
|
|
%% Acquisition flow
|
|
INDEXERS --> PROWLARR
|
|
PROWLARR --> SONARR & RADARR & LIDARR & READARR
|
|
|
|
SONARR --> SAB & QBIT
|
|
RADARR --> SAB & QBIT
|
|
LIDARR --> SAB & QBIT
|
|
READARR --> SAB & QBIT
|
|
|
|
USENET --> SAB
|
|
TORRENT --> QBIT
|
|
|
|
%% Storage flow
|
|
SAB --> MEDIA_TV & MEDIA_MOV & MEDIA_MUS & MEDIA_BOOK
|
|
QBIT --> MEDIA_TV & MEDIA_MOV & MEDIA_MUS & MEDIA_BOOK
|
|
|
|
BAZARR --> MEDIA_TV & MEDIA_MOV
|
|
|
|
%% Streaming flow
|
|
MEDIA_TV & MEDIA_MOV --> PLEX & JELLYFIN
|
|
MEDIA_MUS --> NAVIDROME
|
|
|
|
PLEX --> TAUTULLI
|
|
|
|
%% Client access
|
|
PLEX & JELLYFIN & NAVIDROME --> TV & PHONE & WEB & APPS
|
|
|
|
classDef acquisition fill:#e74c3c,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef storage fill:#3498db,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef streaming fill:#2ecc71,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef client fill:#9b59b6,stroke:#333,stroke-width:2px,color:#fff
|
|
|
|
class PROWLARR,SONARR,RADARR,LIDARR,READARR,BAZARR,SAB,QBIT acquisition
|
|
class MEDIA_TV,MEDIA_MOV,MEDIA_MUS,MEDIA_BOOK storage
|
|
class PLEX,JELLYFIN,NAVIDROME,TAUTULLI streaming
|
|
class TV,PHONE,WEB,APPS client
|
|
```
|
|
|
|
---
|
|
|
|
## 📊 Monitoring Stack Architecture
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph Targets["🎯 Monitored Targets"]
|
|
subgraph Synology["Synology NAS"]
|
|
ATL_SNMP["Atlantis<br/>SNMP"]
|
|
CAL_SNMP["Calypso<br/>SNMP"]
|
|
SET_SNMP["Setillo<br/>SNMP"]
|
|
end
|
|
|
|
subgraph Hosts["Linux Hosts"]
|
|
NODE1["Homelab VM<br/>node_exporter"]
|
|
NODE2["Guava<br/>node_exporter"]
|
|
NODE3["Anubis<br/>node_exporter"]
|
|
end
|
|
|
|
subgraph Containers["Containers"]
|
|
CADV["cAdvisor<br/>Container Metrics"]
|
|
end
|
|
|
|
subgraph Network["Network"]
|
|
BLACK["Blackbox Exporter<br/>HTTP/ICMP Probes"]
|
|
end
|
|
end
|
|
|
|
subgraph Collection["📥 Metric Collection (Homelab VM)"]
|
|
PROM["Prometheus<br/>Time Series DB"]
|
|
SNMP_EXP["SNMP Exporter"]
|
|
end
|
|
|
|
subgraph Visualization["📈 Visualization"]
|
|
GRAFANA["Grafana<br/>Dashboards"]
|
|
end
|
|
|
|
subgraph Alerting["🚨 Alerting"]
|
|
ALERTMGR["Alertmanager"]
|
|
NTFY["ntfy<br/>Push Notifications"]
|
|
UPTIME["Uptime Kuma<br/>Status Page"]
|
|
end
|
|
|
|
%% Collection
|
|
ATL_SNMP & CAL_SNMP & SET_SNMP --> SNMP_EXP
|
|
SNMP_EXP --> PROM
|
|
NODE1 & NODE2 & NODE3 --> PROM
|
|
CADV --> PROM
|
|
BLACK --> PROM
|
|
|
|
%% Visualization
|
|
PROM --> GRAFANA
|
|
PROM --> ALERTMGR
|
|
ALERTMGR --> NTFY
|
|
|
|
%% Uptime Kuma separate
|
|
BLACK -.-> UPTIME
|
|
|
|
classDef target fill:#e67e22,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef collection fill:#3498db,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef viz fill:#2ecc71,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef alert fill:#e74c3c,stroke:#333,stroke-width:2px,color:#fff
|
|
|
|
class ATL_SNMP,CAL_SNMP,SET_SNMP,NODE1,NODE2,NODE3,CADV,BLACK target
|
|
class PROM,SNMP_EXP collection
|
|
class GRAFANA viz
|
|
class ALERTMGR,NTFY,UPTIME alert
|
|
```
|
|
|
|
---
|
|
|
|
## 🔐 Authentication & Security Stack
|
|
|
|
### Complete Authentication Architecture
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph External["🌐 External Access"]
|
|
USERS["👤 Users"]
|
|
CLOUDFLARE["☁️ Cloudflare<br/>DNS/WAF/DDoS"]
|
|
end
|
|
|
|
subgraph Gateway["🚪 Gateway Layer (Calypso)"]
|
|
NPM["🔀 Nginx Proxy Manager<br/>:81/:443<br/>Reverse Proxy + SSL"]
|
|
CFT["🚇 Cloudflare Tunnel<br/>Zero Trust Access"]
|
|
end
|
|
|
|
subgraph AuthLayer["🔐 Authentication Layer (Calypso)"]
|
|
AUTH_SRV["🔐 Authentik Server<br/>:9000"]
|
|
AUTH_PROXY["🛡️ Authentik Outpost<br/>:9444<br/>Forward Auth Proxy"]
|
|
AUTH_WRK["⚙️ Authentik Worker"]
|
|
AUTH_DB["🐘 PostgreSQL"]
|
|
AUTH_RED["🔴 Redis"]
|
|
end
|
|
|
|
subgraph VPN["🔒 VPN Layer"]
|
|
WIREGUARD["🔒 Wireguard<br/>Atlantis :51820"]
|
|
TAILSCALE["🔷 Tailscale<br/>100.x.x.x"]
|
|
HEADSCALE["🌐 Headscale<br/>Calypso :8080"]
|
|
end
|
|
|
|
subgraph DNS["🌐 DNS & Ad Blocking"]
|
|
PIHOLE["🕳️ Pi-hole<br/>Atlantis :53"]
|
|
ADGUARD1["🛡️ AdGuard<br/>Calypso :53"]
|
|
ADGUARD2["🛡️ AdGuard<br/>Setillo :53"]
|
|
end
|
|
|
|
subgraph SecVault["🔑 Secrets Management"]
|
|
VAULT["🔑 Vaultwarden<br/>vault.vish.gg"]
|
|
end
|
|
|
|
subgraph ProtectedServices["🛡️ Protected Services"]
|
|
GRAFANA["📊 Grafana"]
|
|
PAPERLESS["📄 Paperless"]
|
|
IMMICH["📸 Immich"]
|
|
FIREFLY["🔥 Firefly III"]
|
|
ACTUAL["💰 Actual Budget"]
|
|
GITEA["🔧 Gitea"]
|
|
end
|
|
|
|
subgraph PublicServices["🌍 Public/Self-Auth Services"]
|
|
PLEX["📺 Plex"]
|
|
SEAFILE["☁️ Seafile"]
|
|
OST["🚀 OpenSpeedTest"]
|
|
NTFY["📣 ntfy"]
|
|
end
|
|
|
|
%% External flow
|
|
USERS --> CLOUDFLARE
|
|
CLOUDFLARE --> NPM
|
|
CLOUDFLARE --> CFT
|
|
USERS --> TAILSCALE
|
|
|
|
%% NPM to Auth
|
|
NPM -->|"Forward Auth<br/>Header Check"| AUTH_PROXY
|
|
AUTH_PROXY -->|"Validate Session"| AUTH_SRV
|
|
|
|
%% Auth internal
|
|
AUTH_SRV --> AUTH_DB
|
|
AUTH_SRV --> AUTH_RED
|
|
AUTH_WRK --> AUTH_DB
|
|
AUTH_WRK --> AUTH_RED
|
|
|
|
%% Protected services via NPM + Auth
|
|
NPM -->|"✓ Authenticated"| ProtectedServices
|
|
|
|
%% Public services direct
|
|
NPM --> PublicServices
|
|
|
|
%% VPN access
|
|
TAILSCALE --> HEADSCALE
|
|
WIREGUARD --> ProtectedServices
|
|
TAILSCALE --> ProtectedServices
|
|
|
|
%% DNS
|
|
ADGUARD1 -.-> ProtectedServices
|
|
PIHOLE -.-> PublicServices
|
|
|
|
classDef external fill:#e74c3c,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef gateway fill:#f39c12,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef auth fill:#9b59b6,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef dns fill:#1abc9c,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef protected fill:#3498db,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef public fill:#27ae60,stroke:#333,stroke-width:2px,color:#fff
|
|
|
|
class USERS,CLOUDFLARE external
|
|
class NPM,CFT gateway
|
|
class AUTH_SRV,AUTH_PROXY,AUTH_WRK,AUTH_DB,AUTH_RED,VAULT auth
|
|
class PIHOLE,ADGUARD1,ADGUARD2 dns
|
|
class GRAFANA,PAPERLESS,IMMICH,FIREFLY,ACTUAL,GITEA protected
|
|
class PLEX,SEAFILE,OST,NTFY public
|
|
```
|
|
|
|
---
|
|
|
|
### Authentik SSO Flow (Detailed)
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
autonumber
|
|
participant U as 👤 User
|
|
participant CF as ☁️ Cloudflare
|
|
participant NPM as 🔀 NPM (Calypso)
|
|
participant OUT as 🛡️ Outpost (Calypso)
|
|
participant AUTH as 🔐 Authentik (Calypso)
|
|
participant APP as 📱 Application
|
|
|
|
U->>CF: Request app.vish.gg
|
|
CF->>NPM: Forward (HTTPS)
|
|
NPM->>OUT: Forward Auth Request<br/>(/outpost.goauthentik.io/auth/nginx)
|
|
|
|
alt No Valid Session
|
|
OUT->>AUTH: Check Session
|
|
AUTH-->>OUT: No Session
|
|
OUT-->>NPM: 401 Unauthorized
|
|
NPM-->>U: Redirect to sso.vish.gg/flows/default-authentication/
|
|
U->>AUTH: Login Page
|
|
U->>AUTH: Submit Credentials + 2FA
|
|
AUTH->>AUTH: Validate
|
|
AUTH-->>U: Set Cookie + Redirect to app
|
|
U->>NPM: Retry with Session Cookie
|
|
NPM->>OUT: Forward Auth (with cookie)
|
|
end
|
|
|
|
OUT->>AUTH: Validate Session
|
|
AUTH-->>OUT: Valid ✓
|
|
OUT-->>NPM: 200 OK + Headers<br/>(X-authentik-username, X-authentik-email)
|
|
NPM->>APP: Proxy Request (with auth headers)
|
|
APP-->>U: Response
|
|
```
|
|
|
|
---
|
|
|
|
### NPM Proxy Host Configuration
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph NPM["🔀 Nginx Proxy Manager (Calypso :81)"]
|
|
subgraph ProxyHosts["Proxy Hosts"]
|
|
PH1["sso.vish.gg → Calypso:9000"]
|
|
PH2["git.vish.gg → Calypso:3000"]
|
|
PH3["docs.vish.gg → Atlantis:8088"]
|
|
PH4["photos.vish.gg → Calypso:2283"]
|
|
PH5["gf.vish.gg → Atlantis:3000"]
|
|
PH6["actual.vish.gg → Calypso:5006"]
|
|
PH7["ff.vish.gg → Calypso:8888"]
|
|
PH8["plex.vish.gg → Atlantis:32400"]
|
|
PH9["sf.vish.gg → Calypso:8092"]
|
|
PH10["ntfy.vish.gg → Homelab:7080"]
|
|
PH11["rackula.vish.gg → Calypso:4999"]
|
|
end
|
|
|
|
subgraph SSL["SSL Certificates"]
|
|
WILD["*.vish.gg<br/>Cloudflare DNS Challenge"]
|
|
end
|
|
|
|
subgraph AccessControl["Access Control"]
|
|
AUTH_LOC["Authentik Forward Auth<br/>Location: /outpost.goauthentik.io"]
|
|
end
|
|
end
|
|
|
|
subgraph Services["Backend Services"]
|
|
direction LR
|
|
S1["Authentik"]
|
|
S2["Gitea"]
|
|
S3["Paperless"]
|
|
S4["Immich"]
|
|
S5["Grafana"]
|
|
S6["Actual"]
|
|
S7["Firefly"]
|
|
S8["Plex"]
|
|
S9["Seafile"]
|
|
S10["ntfy"]
|
|
S11["Rackula"]
|
|
end
|
|
|
|
PH1 --> S1
|
|
PH2 --> S2
|
|
PH3 --> S3
|
|
PH4 --> S4
|
|
PH5 --> S5
|
|
PH6 --> S6
|
|
PH7 --> S7
|
|
PH8 --> S8
|
|
PH9 --> S9
|
|
PH10 --> S10
|
|
PH11 --> S11
|
|
```
|
|
|
|
---
|
|
|
|
### Services Protected by Authentik
|
|
|
|
| Domain | Service | Host | Auth Type | Notes |
|
|
|--------|---------|------|-----------|-------|
|
|
| `sso.vish.gg` | Authentik | Calypso | - | Identity Provider |
|
|
| `git.vish.gg` | Gitea | Calypso | OAuth2/OIDC | Source Control |
|
|
| `gf.vish.gg` | Grafana | Atlantis | OAuth2/OIDC | Monitoring |
|
|
| `docs.vish.gg` | Paperless-NGX | Atlantis | Forward Auth | Documents |
|
|
| `photos.vish.gg` | Immich | Calypso | Forward Auth | Photos |
|
|
| `actual.vish.gg` | Actual Budget | Calypso | Forward Auth | Finance |
|
|
| `ff.vish.gg` | Firefly III | Calypso | Forward Auth | Finance |
|
|
| `rackula.vish.gg` | Rackula | Calypso | Forward Auth | Rack Diagram |
|
|
|
|
### Services NOT Protected (Public/Self-Auth)
|
|
|
|
| Domain | Service | Host | Reason |
|
|
|--------|---------|------|--------|
|
|
| `plex.vish.gg` | Plex | Atlantis | Has Plex Auth |
|
|
| `sf.vish.gg` | Seafile | Calypso | Has built-in auth + share links |
|
|
| `ntfy.vish.gg` | ntfy | Homelab | Has built-in auth + public topics |
|
|
| `ost.vish.gg` | OpenSpeedTest | Calypso | Public utility |
|
|
|
|
---
|
|
|
|
### Authentik Forward Auth Setup (NPM)
|
|
|
|
To protect a service with Authentik Forward Auth in NPM:
|
|
|
|
1. **Create Provider in Authentik**:
|
|
- Type: Proxy Provider
|
|
- External Host: `https://app.vish.gg`
|
|
- Mode: Forward auth (single application)
|
|
|
|
2. **Create Application in Authentik**:
|
|
- Link to the provider
|
|
- Set policies for access control
|
|
|
|
3. **Create Outpost in Authentik**:
|
|
- Type: Proxy
|
|
- Include the application
|
|
|
|
4. **Configure NPM Proxy Host**:
|
|
```nginx
|
|
# Custom Nginx Configuration (Advanced tab)
|
|
|
|
# Authentik Forward Auth
|
|
location /outpost.goauthentik.io {
|
|
proxy_pass http://calypso.vish.local:9444/outpost.goauthentik.io;
|
|
proxy_set_header Host $host;
|
|
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
|
|
add_header Set-Cookie $auth_cookie;
|
|
auth_request_set $auth_cookie $upstream_http_set_cookie;
|
|
proxy_pass_request_body off;
|
|
proxy_set_header Content-Length "";
|
|
}
|
|
|
|
location / {
|
|
auth_request /outpost.goauthentik.io/auth/nginx;
|
|
error_page 401 = @goauthentik_proxy_signin;
|
|
auth_request_set $auth_cookie $upstream_http_set_cookie;
|
|
add_header Set-Cookie $auth_cookie;
|
|
|
|
# Forward auth headers to application
|
|
auth_request_set $authentik_username $upstream_http_x_authentik_username;
|
|
auth_request_set $authentik_email $upstream_http_x_authentik_email;
|
|
proxy_set_header X-authentik-username $authentik_username;
|
|
proxy_set_header X-authentik-email $authentik_email;
|
|
|
|
proxy_pass http://backend;
|
|
}
|
|
|
|
location @goauthentik_proxy_signin {
|
|
internal;
|
|
add_header Set-Cookie $auth_cookie;
|
|
return 302 /outpost.goauthentik.io/start?rd=$request_uri;
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 📝 ASCII Service Distribution by Host
|
|
|
|
```
|
|
╔══════════════════════════════════════════════════════════════════════════════════════════╗
|
|
║ SERVICE DISTRIBUTION BY HOST ║
|
|
╚══════════════════════════════════════════════════════════════════════════════════════════╝
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🏛️ ATLANTIS (55 Services) - Primary Hub │
|
|
│ ═══════════════════════════════════════════════════════════════════════════════════════│
|
|
│ │
|
|
│ 📺 Media 📊 Monitoring 🔐 Security 🛠️ Infrastructure │
|
|
│ ───────────── ───────────── ───────────── ───────────────── │
|
|
│ • Plex • Grafana • Vaultwarden • Portainer │
|
|
│ • Jellyfin • Prometheus • Wireguard • Nginx Proxy Mgr │
|
|
│ • Immich • Uptime Kuma • Pi-hole • DokuWiki │
|
|
│ • Tautulli • SNMP Exporter • Dozzle │
|
|
│ • Arr Suite (7) • Blackbox Exp • Watchtower │
|
|
│ • Navidrome │
|
|
│ │
|
|
│ 💬 Communication 📝 Productivity 🎮 Other │
|
|
│ ───────────── ───────────── ───────────── │
|
|
│ • Matrix Synapse • Paperless-NGX • IT-Tools │
|
|
│ • Mastodon • Firefly III • Stirling PDF │
|
|
│ • Joplin Server • Documenso • YouTube DL │
|
|
│ • Netbox │
|
|
└─────────────────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🏢 CALYPSO (17 Services) - Development & Backup │
|
|
│ ═══════════════════════════════════════════════════════════════════════════════════════│
|
|
│ │
|
|
│ 💻 Development 💰 Finance 📦 Infrastructure 📸 Media │
|
|
│ ───────────── ───────────── ───────────── ───────────── │
|
|
│ • Gitea • Firefly III • APT-Cacher-NG • Immich (backup) │
|
|
│ • Reactive Resume • Actual Budget • Prometheus • Seafile │
|
|
│ • Rustdesk • Wireguard │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 💻 HOMELAB VM (36 Services) - Experimentation │
|
|
│ ═══════════════════════════════════════════════════════════════════════════════════════│
|
|
│ │
|
|
│ 🔗 URL Services 🎮 Gaming 📊 Monitoring 🔧 Utilities │
|
|
│ ───────────── ───────────── ───────────── ───────────── │
|
|
│ • Shlink • Satisfactory • Prometheus Hub • Archivebox │
|
|
│ • ntfy • Minecraft • node_exporter • WebCheck │
|
|
│ • Hoarder • L4D2 • Redlib │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🌐 CONCORD NUC (9 Services) - Edge/IoT │
|
|
│ ═══════════════════════════════════════════════════════════════════════════════════════│
|
|
│ │
|
|
│ 🏠 Home Automation 📺 Media 🎵 Music 🔧 Network │
|
|
│ ───────────── ───────────── ───────────── ───────────── │
|
|
│ • Home Assistant • Plex • YourSpotify • AdGuard Home │
|
|
│ • Matter Server • Invidious • Piped • Wireguard │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 💻 GUAVA (12+ Services) - TrueNAS Scale (Ryzen 5 8600G) │
|
|
│ ═══════════════════════════════════════════════════════════════════════════════════════│
|
|
│ │
|
|
│ 🧠 AI/ML 📺 Media 🔧 Development 🌐 Network │
|
|
│ ───────────── ───────────── ───────────── ───────────── │
|
|
│ • Ollama • Jellyfin • Gitea • WireGuard │
|
|
│ • LlamaGPT • Fenrus • CoCalc • Tailscale │
|
|
│ • Fasten Health • Portainer │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🌵 SETILLO (4 Services) - Tucson Remote │
|
|
│ ═══════════════════════════════════════════════════════════════════════════════════════│
|
|
│ │
|
|
│ 📊 Monitoring 🌐 DNS │
|
|
│ ───────────── ───────────── │
|
|
│ • Prometheus • AdGuard Home │
|
|
│ • SNMP Exporter • Syncthing │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
╔════════════════════════════════════════════════════════════════════════════════════════╗
|
|
║ SERVICE COUNT SUMMARY ║
|
|
║ ═════════════════════ ║
|
|
║ Atlantis: 51 containers │ Calypso: 54 containers ║
|
|
║ Homelab VM: 30 containers │ Concord NUC: 18 containers ║
|
|
║ Guava: 12+ containers │ matrix-ubuntu: 6 containers ║
|
|
║ Setillo: 4 containers │ RPi nodes: 2 containers ║
|
|
║ Seattle VPS: 7 containers │ ║
|
|
║ ──────────────────────────────────────────────────────────────────────────────────────║
|
|
║ TOTAL: ~184 containers across 9 active hosts ║
|
|
║ (Per Portainer: 159 containers across 5 managed endpoints) ║
|
|
╚════════════════════════════════════════════════════════════════════════════════════════╝
|
|
```
|
|
|
|
---
|
|
|
|
## 🔗 Related Diagrams
|
|
- [Network Topology](network-topology.md) - How hosts connect
|
|
- [Storage Topology](storage-topology.md) - Where data lives
|
|
- [Tailscale Mesh](tailscale-mesh.md) - Cross-location access
|
|
|
|
---
|
|
|
|
## 💬 Communication Stack Architecture
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph Internet["☁️ Internet / Federation"]
|
|
FEDI["Fediverse<br/>(ActivityPub)"]
|
|
MATRIX_FED["Matrix<br/>Federation"]
|
|
WEBRTC["WebRTC<br/>Voice/Video"]
|
|
end
|
|
|
|
subgraph Cloudflare["🛡️ Cloudflare"]
|
|
CF_PROXY["Cloudflare<br/>Proxy/WAF"]
|
|
CF_TUNNEL["Cloudflare<br/>Tunnel"]
|
|
end
|
|
|
|
subgraph MatrixUbuntuVM["🐧 Matrix-Ubuntu VM (Atlantis)"]
|
|
subgraph Mastodon["🐘 Mastodon Stack"]
|
|
MASTO_WEB["Mastodon Web<br/>:3000"]
|
|
MASTO_STREAM["Mastodon Streaming<br/>:4000"]
|
|
MASTO_SIDEKIQ["Sidekiq<br/>Background Jobs"]
|
|
end
|
|
|
|
subgraph Matrix["🔐 Matrix Stack"]
|
|
SYNAPSE["Synapse<br/>:8008 / :8018"]
|
|
ELEMENT["Element Web<br/>Client"]
|
|
COTURN["Coturn<br/>TURN Server<br/>:3478"]
|
|
end
|
|
|
|
subgraph Mattermost["💬 Mattermost"]
|
|
MM_APP["Mattermost<br/>:8065"]
|
|
end
|
|
|
|
subgraph SharedDB["🗄️ Shared Services"]
|
|
POSTGRES["PostgreSQL<br/>:5432"]
|
|
REDIS["Redis<br/>:6379"]
|
|
end
|
|
|
|
NGINX_VM["Nginx<br/>Reverse Proxy"]
|
|
end
|
|
|
|
subgraph Atlantis["🏛️ Atlantis NAS"]
|
|
subgraph JitsiStack["📹 Jitsi Meet"]
|
|
JITSI_WEB["Jitsi Web"]
|
|
JITSI_JVB["Jitsi Video Bridge"]
|
|
JITSI_PROSODY["Prosody XMPP"]
|
|
end
|
|
|
|
subgraph Vaultwarden["🔑 Vaultwarden"]
|
|
VW["Vaultwarden<br/>Password Manager"]
|
|
end
|
|
|
|
subgraph Joplin["📝 Joplin"]
|
|
JOPLIN_SRV["Joplin Server"]
|
|
end
|
|
end
|
|
|
|
subgraph Clients["📱 Clients"]
|
|
BROWSER["Web Browsers"]
|
|
MOBILE["Mobile Apps"]
|
|
DESKTOP["Desktop Apps"]
|
|
end
|
|
|
|
%% External connections
|
|
FEDI <--> CF_PROXY
|
|
MATRIX_FED <--> CF_PROXY
|
|
WEBRTC <--> COTURN
|
|
|
|
%% Cloudflare to services
|
|
CF_PROXY --> NGINX_VM
|
|
CF_TUNNEL --> NGINX_VM
|
|
|
|
%% Nginx routing
|
|
NGINX_VM --> MASTO_WEB & MASTO_STREAM
|
|
NGINX_VM --> SYNAPSE & ELEMENT
|
|
NGINX_VM --> MM_APP
|
|
|
|
%% Database connections
|
|
MASTO_WEB & MASTO_SIDEKIQ --> POSTGRES & REDIS
|
|
SYNAPSE --> POSTGRES
|
|
MM_APP --> POSTGRES
|
|
|
|
%% Client access
|
|
BROWSER & MOBILE & DESKTOP --> CF_PROXY
|
|
BROWSER & MOBILE & DESKTOP --> JITSI_WEB
|
|
BROWSER & MOBILE & DESKTOP --> VW
|
|
BROWSER & MOBILE & DESKTOP --> JOPLIN_SRV
|
|
|
|
classDef mastodon fill:#6364FF,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef matrix fill:#0DBD8B,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef mattermost fill:#0058CC,stroke:#333,stroke-width:2px,color:#fff
|
|
classDef infra fill:#e67e22,stroke:#333,stroke-width:2px,color:#fff
|
|
|
|
class MASTO_WEB,MASTO_STREAM,MASTO_SIDEKIQ mastodon
|
|
class SYNAPSE,ELEMENT,COTURN matrix
|
|
class MM_APP mattermost
|
|
class POSTGRES,REDIS,NGINX_VM infra
|
|
```
|
|
|
|
### Communication Services Summary
|
|
|
|
| Service | Domain | Protocol | Purpose |
|
|
|---------|--------|----------|---------|
|
|
| **Mastodon** | mastodon.vish.gg | ActivityPub | Fediverse microblogging |
|
|
| **Matrix (Primary)** | mx.vish.gg | Matrix | Federated chat |
|
|
| **Matrix (Legacy)** | matrix.thevish.io | Matrix | Legacy homeserver |
|
|
| **Mattermost** | mm.crista.love | Proprietary | Team collaboration |
|
|
| **Jitsi Meet** | meet.vish.gg | WebRTC | Video conferencing |
|
|
| **Joplin** | joplin.vish.gg | Joplin Sync | Note synchronization |
|
|
| **Vaultwarden** | vault.vish.gg | Bitwarden | Password management |
|
|
|
|
### Deployment Scripts
|
|
|
|
| Script | Location | Description |
|
|
|--------|----------|-------------|
|
|
| Mastodon Install | [mastodon-production/](../mastodon-production/) | Bare metal & Docker deployment |
|
|
| Matrix Install | [matrix-element/](../matrix-element/) | Synapse + Element + TURN |
|
|
| Mattermost Install | [mattermost-production/](../mattermost-production/) | Docker deployment |
|
|
| VM Config | [matrix-ubuntu-vm/](../matrix-ubuntu-vm/) | Complete VM configuration |
|
|
|
|
---
|
|
|
|
## 🔄 CI/CD Pipeline Architecture
|
|
|
|
### Git Repository Mirroring
|
|
|
|
The homelab repository uses Gitea Actions for automated CI/CD, including sanitized public mirroring.
|
|
|
|
```mermaid
|
|
graph LR
|
|
subgraph Development["💻 Development"]
|
|
DEV["Developer<br/>Pushes Code"]
|
|
end
|
|
|
|
subgraph Gitea["🔧 Gitea (Calypso)"]
|
|
PRIVATE["🔒 Private Repo<br/>homelab"]
|
|
PUBLIC["🌐 Public Repo<br/>homelab-optimized"]
|
|
RUNNER["🏃 Gitea Runner<br/>(Calypso)"]
|
|
end
|
|
|
|
subgraph Workflow["⚙️ CI/CD Workflow"]
|
|
CHECKOUT["📥 Checkout Code"]
|
|
SANITIZE["🧹 Sanitize<br/>Remove Secrets"]
|
|
PUSH["📤 Force Push<br/>Fresh History"]
|
|
end
|
|
|
|
subgraph Deployment["🚀 Deployment"]
|
|
ANSIBLE["📋 Ansible<br/>Multi-host"]
|
|
PORTAINER["🐳 Portainer<br/>5 Endpoints"]
|
|
end
|
|
|
|
DEV -->|"git push"| PRIVATE
|
|
PRIVATE -->|"Triggers"| RUNNER
|
|
RUNNER --> CHECKOUT
|
|
CHECKOUT --> SANITIZE
|
|
SANITIZE --> PUSH
|
|
PUSH --> PUBLIC
|
|
|
|
PRIVATE --> ANSIBLE
|
|
ANSIBLE --> PORTAINER
|
|
```
|
|
|
|
### Sanitization Process
|
|
|
|
The sanitization script removes sensitive data before public mirroring:
|
|
|
|
| Removed | Pattern | Example |
|
|
|---------|---------|---------|
|
|
| Passwords | `password:`, `PASS=` | `password: "REDACTED_PASSWORD" |
|
|
| API Keys | `api_key:`, `API_KEY=` | `api_key: REDACTED_API_KEY` |
|
|
| Tokens | `token:`, `TOKEN=` | `token: REDACTED_TOKEN` |
|
|
| Secrets | `secret:`, `SECRET=` | `secret: REDACTED_SECRET` |
|
|
| Private Keys | `-----BEGIN.*KEY-----` | File removed |
|
|
| SSH Keys | `id_rsa`, `id_ed25519` | File removed |
|
|
| Personal Emails | `*@gmail.com`, `*@*.com` | `REDACTED_EMAIL@example.com` |
|
|
| JWT Secrets | `JWT_SECRET=` | `JWT_SECRET=REDACTED` |
|
|
|
|
### Gitea Runner Setup
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph Calypso["🌊 Calypso (DS723+)"]
|
|
GITEA["🔧 Gitea Server<br/>:3000"]
|
|
RUNNER["🏃 Gitea Runner<br/>act_runner:latest"]
|
|
DOCKER["🐳 Docker Socket"]
|
|
end
|
|
|
|
GITEA -->|"Workflow Dispatch"| RUNNER
|
|
RUNNER -->|"docker.sock"| DOCKER
|
|
DOCKER -->|"Spawn Containers"| RUNNER
|
|
```
|
|
|
|
**Runner Configuration:**
|
|
- Image: `gitea/act_runner:latest`
|
|
- Labels: `ubuntu-latest`, `ubuntu-22.04`, `python`
|
|
- Location: Portainer stack on Calypso
|
|
- Trigger: Push to main branch
|
|
|
|
### Ansible Automation
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph Control["📋 Ansible Control"]
|
|
SITE["site.yml<br/>Master Playbook"]
|
|
INV["inventory.yml<br/>13 Hosts"]
|
|
ROLES["Roles<br/>docker_stack, directory_setup"]
|
|
end
|
|
|
|
subgraph Hosts["🖥️ Target Hosts"]
|
|
SYN["Synology<br/>Atlantis, Calypso, Setillo"]
|
|
VMS["VMs<br/>Homelab, matrix-ubuntu"]
|
|
PHYS["Physical<br/>Guava, NUC, Shinku-Ryuu"]
|
|
EDGE["Edge<br/>RPi5, Jellyfish"]
|
|
CLOUD["Cloud<br/>Seattle VPS"]
|
|
end
|
|
|
|
SITE --> INV
|
|
INV --> SYN
|
|
INV --> VMS
|
|
INV --> PHYS
|
|
INV --> EDGE
|
|
INV --> CLOUD
|
|
```
|
|
|
|
**Ansible Commands:**
|
|
```bash
|
|
# Deploy everything
|
|
ansible-playbook site.yml
|
|
|
|
# Deploy to specific host
|
|
ansible-playbook site.yml --limit atlantis
|
|
|
|
# Deploy by category
|
|
ansible-playbook site.yml --tags synology
|
|
|
|
# Check status
|
|
ansible-playbook playbooks/common/status.yml
|
|
```
|
|
|
|
---
|
|
|
|
## 🔗 Related Diagrams
|
|
- [Network Topology](network-topology.md) - How hosts connect
|
|
- [Storage Topology](storage-topology.md) - Where data lives
|
|
- [Tailscale Mesh](tailscale-mesh.md) - Cross-location access
|