fx-test/fluxer/SELF_HOSTING.md

Fluxer — Self-Hosting & VM Rebuild Guide

This document covers how to spin up the Fluxer dev instance from scratch on a new VM. The running reference deployment is at st.vish.gg (seattle VM, 66.94.122.170 / 100.82.197.124).

---

Architecture overview

All services run as Docker containers via dev/compose.yaml, connected on a shared Docker network (fluxer-shared). Caddy acts as the internal reverse proxy, routing all traffic on port 8088 to the appropriate service. Cloudflared tunnels external HTTPS traffic (no open inbound ports required).

Internet
  └── Cloudflare Tunnel (cloudflared)
        └── Caddy :8088
              ├── /api/*         → api:8080        (Node.js REST API)
              ├── /gateway/*     → gateway:8080     (Erlang WebSocket gateway)
              ├── /media/*       → media:8080       (media proxy)
              ├── /admin/*       → admin:8080       (Elixir admin panel)
              ├── /marketing/*   → marketing:8080   (Elixir marketing site)
              ├── /metrics/*     → metrics:8080     (Rust metrics service)
              ├── /livekit/*     → livekit:7880     (LiveKit voice/video)
              ├── /s3/*          → minio:9000       (S3-compatible object storage)
              └── /*             → /app/dist        (built frontend SPA)

Services

Container	Image/Build	Purpose
caddy	caddy:2	Internal reverse proxy
cloudflared	cloudflare/cloudflared	Tunnel (no inbound ports)
api	node:24-bookworm-slim	REST API (tsx watch)
worker	node:24-bookworm-slim	Background job worker
gateway	erlang:28-slim	WebSocket gateway (compiled with rebar3)
media	build from fluxer_media_proxy/	Media proxy
admin	build from fluxer_admin/	Admin panel
marketing	build from fluxer_marketing/	Marketing site
docs	node:24-bookworm-slim	Docs site
metrics	build from fluxer_metrics/	Metrics aggregator
postgres	postgres:17	Primary database
cassandra	scylladb/scylla:latest	Time-series / message data
redis	valkey/valkey:latest	Cache / pub-sub
minio	minio/minio	S3-compatible object storage
meilisearch	getmeili/meilisearch:v1.25.0	Full-text search
clamav	clamav/clamav:latest	Virus scanning (can be disabled)
livekit	livekit/livekit-server:latest	Voice & video
cassandra-migrate	debian:bookworm-slim	One-shot DB migration runner

---

Prerequisites

• Debian/Ubuntu VM (tested on Debian 12 bookworm)
• Docker + Docker Compose plugin installed
• git installed
• A Cloudflare tunnel token (from Cloudflare Zero Trust dashboard)
• Domain pointed at Cloudflare (e.g. st.vish.gg)
• At least 4 vCPU / 8GB RAM recommended (ScyllaDB + ClamAV are hungry)

Install Docker (if not present)

curl -fsSL https://get.docker.com | sh
systemctl enable --now docker

---

Step-by-step rebuild

1. Clone the repo

git clone https://github.com/fluxerapp/fluxer /root/fluxer
cd /root/fluxer

2. Create the shared Docker network

Must be created before starting any containers — it's declared external: true in the compose file.

docker network create fluxer-shared

3. Build the frontend app

Caddy serves the pre-built SPA from fluxer_app/dist/. This must be built before starting Caddy, otherwise it will mount an empty directory and serve nothing.

# Install pnpm
corepack enable pnpm

# Install workspace deps
pnpm install

# Build the app (outputs to fluxer_app/dist/)
cd fluxer_app
pnpm build
cd ..

Note: The build requires Node 22+ and pnpm. If the VM doesn't have Node installed globally, run the build inside a container:

docker run --rm -v $(pwd):/workspace -w /workspace/fluxer_app node:24-bookworm-slim \
  bash -lc "corepack enable pnpm && CI=true pnpm install && pnpm build"

4. Build the cassandra-migrate binary

The Cassandra migration runner is a Rust binary that must be pre-compiled.

# Install Rust if needed
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source ~/.cargo/env

# Build
cd scripts/cassandra-migrate
cargo build --release
cd ../..

The compose file expects the binary at scripts/cassandra-migrate/target/release/cassandra-migrate.

5. Set up the environment file

cp dev/.env.example dev/.env

Edit dev/.env and fill in:

Variable	What to set
FLUXER_APP_ENDPOINT	Your public URL e.g. https://st.vish.gg
FLUXER_API_PUBLIC_ENDPOINT	https://st.vish.gg/api
FLUXER_GATEWAY_ENDPOINT	wss://st.vish.gg/gateway
FLUXER_MARKETING_ENDPOINT	https://st.vish.gg
FLUXER_ADMIN_ENDPOINT	https://st.vish.gg/admin
FLUXER_MEDIA_ENDPOINT	https://st.vish.gg/media
FLUXER_INVITE_ENDPOINT	https://st.vish.gg
CLOUDFLARE_TUNNEL_TOKEN	Token from Cloudflare Zero Trust dashboard
SECRET_KEY_BASE	Generate: openssl rand -hex 64
GATEWAY_RPC_SECRET	Generate: openssl rand -hex 32
GATEWAY_ADMIN_SECRET	Generate: openssl rand -hex 32
MEDIA_PROXY_SECRET_KEY	Generate: openssl rand -hex 32
SUDO_MODE_SECRET	Generate: openssl rand -hex 32
LIVEKIT_API_KEY	Generate: openssl rand -hex 16
LIVEKIT_API_SECRET	Generate: openssl rand -hex 32
VAPID_PUBLIC_KEY	Generate with web-push: npx web-push generate-vapid-keys
VAPID_PRIVATE_KEY	(paired with above)
ADMIN_OAUTH2_CLIENT_ID	OAuth2 app ID (from Fluxer admin panel after first boot)
ADMIN_OAUTH2_CLIENT_SECRET	OAuth2 secret
PASSKEY_RP_ID	Your domain e.g. st.vish.gg
PASSKEY_ALLOWED_ORIGINS	https://st.vish.gg

Everything else can stay at its default dev value on first boot.

6. Set up the LiveKit config

cp dev/livekit.yaml dev/livekit.yaml

Edit dev/livekit.yaml — replace the API key/secret with the values you set in .env:

keys:
  "<LIVEKIT_API_KEY>": "<LIVEKIT_API_SECRET>"

webhook:
  api_key: "<LIVEKIT_API_KEY>"
  urls:
    - "http://api:8080/webhooks/livekit"

7. Start the stack

cd /root/fluxer/dev
docker compose up -d

On first boot, cassandra-migrate will run automatically and apply schema migrations. ScyllaDB takes ~90 seconds to become healthy before migrations run.

Watch progress:

docker compose logs -f cassandra-migrate
docker compose logs -f api

8. Verify everything is up

docker compose ps

Expected running containers: caddy, cloudflared, api, worker, gateway, media, admin, marketing, docs, metrics, clamav, meilisearch, redis, minio, livekit

Expected exited (one-shot): cassandra-migrate, minio-setup

The app should be accessible at your configured domain via the Cloudflare tunnel.

---

Managing the stack

Start / stop

cd /root/fluxer/dev

# Start all
docker compose up -d

# Stop all (keep volumes)
docker compose down

# Stop and wipe all data
docker compose down -v

View logs

# All services
docker compose logs -f

# Specific service
docker compose logs -f api
docker compose logs -f gateway

Rebuild a specific service after code changes

# Rebuild and restart one service
docker compose up -d --build media

# Rebuild frontend app and restart caddy
cd /root/fluxer
pnpm --filter fluxer_app build
docker compose -f dev/compose.yaml restart caddy

Update from upstream

cd /root/fluxer
git pull origin main

# Rebuild frontend
pnpm install
pnpm --filter fluxer_app build

# Restart services that auto-install deps (api, worker, docs)
docker compose -f dev/compose.yaml restart api worker docs

# Rebuild services with Dockerfiles
docker compose -f dev/compose.yaml up -d --build media admin marketing metrics

---

Configuration reference

Key files

File	Purpose
dev/compose.yaml	Main docker compose for the dev stack
dev/.env	All environment variables (not committed — see .env.example)
dev/Caddyfile.dev	Caddy reverse proxy routing config
dev/livekit.yaml	LiveKit server config (API keys, webhook URL)

Port mappings (host)

Port	Service
8088	Caddy (all traffic enters here, then routed internally)
7880	LiveKit HTTP
7882/udp	LiveKit RTC
7999/udp	LiveKit RTC
9042	ScyllaDB CQL (direct access)
8123	ClickHouse (only with --profile clickhouse)

All other services communicate internally over fluxer-shared Docker network.

Optional profiles

# Enable ClickHouse-backed metrics
docker compose --profile clickhouse up -d

# Enable search (meilisearch is included by default in dev)
docker compose --profile search up -d

---

Cloudflare tunnel setup

1. Go to Cloudflare Zero Trust → Networks → Tunnels → Create tunnel
2. Name it (e.g. fluxer-seattle)
3. Copy the tunnel token → set as CLOUDFLARE_TUNNEL_TOKEN in dev/.env
4. Add a public hostname:
   • Subdomain: st (or your chosen subdomain)
   • Domain: vish.gg
   • Service: http://localhost:8088
5. The cloudflared container will connect automatically on docker compose up

---

Troubleshooting

Gateway won't start

The Erlang gateway compiles from source on first start (rebar3 compile) — this takes 2-5 minutes. Watch with docker compose logs -f gateway. If it exits, check for missing deps or compilation errors.

ScyllaDB/Cassandra migration fails

ScyllaDB needs ~90 seconds to fully initialise. The cassandra-migrate container has a sleep 30 but on slow machines it may still fail. Restart it manually:

docker compose run --rm cassandra-migrate

API exits immediately

Almost always a missing or malformed .env value. Check:

docker compose logs api | head -30

Common causes: DATABASE_URL unreachable, missing SECRET_KEY_BASE, malformed REDIS_URL.

Frontend shows blank page

The fluxer_app/dist/ directory is empty — the app hasn't been built yet. Run the build step (Step 3 above) then restart Caddy:

docker compose restart caddy

ClamAV takes forever to start

ClamAV downloads its virus definition database on first start (~500MB). This is normal. It won't be marked healthy until the download completes (~5-10 min on first boot). ClamAV can be disabled by setting CLAMAV_ENABLED=false in .env — the other services don't depend on it.

---

Seattle reference deployment

The running instance at st.vish.gg uses this exact setup. Key details:

Property	Value
Host	Seattle VM (66.94.122.170 / Tailscale 100.82.197.124)
SSH	ssh root@seattle or ssh root@seattle-tailscale
Install path	/root/fluxer/
Compose path	/root/fluxer/dev/
Domain	st.vish.gg
Tunnel	Cloudflare Zero Trust (cloudflared)
AUTO_JOIN_INVITE_CODE	FRIENDS (anyone with this code can register)

To check status on seattle:

ssh seattle-tailscale "cd /root/fluxer/dev && docker compose ps"
ssh seattle-tailscale "cd /root/fluxer/dev && docker compose logs --tail 20 api"