fx-test/fluxer/SELF_HOSTING.md

# Fluxer — Self-Hosting & VM Rebuild Guide

This document covers how to spin up the Fluxer dev instance from scratch on a new VM.
The running reference deployment is at `st.vish.gg` (seattle VM, `66.94.122.170` / `100.82.197.124`).

---

## Architecture overview

All services run as Docker containers via `dev/compose.yaml`, connected on a shared Docker network (`fluxer-shared`).
Caddy acts as the internal reverse proxy, routing all traffic on port `8088` to the appropriate service.
Cloudflared tunnels external HTTPS traffic (no open inbound ports required).

```
Internet
  └── Cloudflare Tunnel (cloudflared)
        └── Caddy :8088
              ├── /api/*         → api:8080        (Node.js REST API)
              ├── /gateway/*     → gateway:8080     (Erlang WebSocket gateway)
              ├── /media/*       → media:8080       (media proxy)
              ├── /admin/*       → admin:8080       (Elixir admin panel)
              ├── /marketing/*   → marketing:8080   (Elixir marketing site)
              ├── /metrics/*     → metrics:8080     (Rust metrics service)
              ├── /livekit/*     → livekit:7880     (LiveKit voice/video)
              ├── /s3/*          → minio:9000       (S3-compatible object storage)
              └── /*             → /app/dist        (built frontend SPA)
```

### Services

| Container | Image/Build | Purpose |
|-----------|-------------|---------|
| `caddy` | `caddy:2` | Internal reverse proxy |
| `cloudflared` | `cloudflare/cloudflared` | Tunnel (no inbound ports) |
| `api` | `node:24-bookworm-slim` | REST API (`tsx watch`) |
| `worker` | `node:24-bookworm-slim` | Background job worker |
| `gateway` | `erlang:28-slim` | WebSocket gateway (compiled with rebar3) |
| `media` | build from `fluxer_media_proxy/` | Media proxy |
| `admin` | build from `fluxer_admin/` | Admin panel |
| `marketing` | build from `fluxer_marketing/` | Marketing site |
| `docs` | `node:24-bookworm-slim` | Docs site |
| `metrics` | build from `fluxer_metrics/` | Metrics aggregator |
| `postgres` | `postgres:17` | Primary database |
| `cassandra` | `scylladb/scylla:latest` | Time-series / message data |
| `redis` | `valkey/valkey:latest` | Cache / pub-sub |
| `minio` | `minio/minio` | S3-compatible object storage |
| `meilisearch` | `getmeili/meilisearch:v1.25.0` | Full-text search |
| `clamav` | `clamav/clamav:latest` | Virus scanning (can be disabled) |
| `livekit` | `livekit/livekit-server:latest` | Voice & video |
| `cassandra-migrate` | `debian:bookworm-slim` | One-shot DB migration runner |

---

## Prerequisites

- Debian/Ubuntu VM (tested on Debian 12 bookworm)
- Docker + Docker Compose plugin installed
- `git` installed
- A Cloudflare tunnel token (from Cloudflare Zero Trust dashboard)
- Domain pointed at Cloudflare (e.g. `st.vish.gg`)
- At least 4 vCPU / 8GB RAM recommended (ScyllaDB + ClamAV are hungry)

### Install Docker (if not present)

```bash
curl -fsSL https://get.docker.com | sh
systemctl enable --now docker
```

---

## Step-by-step rebuild

### 1. Clone the repo

```bash
git clone https://github.com/fluxerapp/fluxer /root/fluxer
cd /root/fluxer
```

### 2. Create the shared Docker network

Must be created before starting any containers — it's declared `external: true` in the compose file.

```bash
docker network create fluxer-shared
```

### 3. Build the frontend app

Caddy serves the pre-built SPA from `fluxer_app/dist/`. This must be built before starting Caddy,
otherwise it will mount an empty directory and serve nothing.

```bash
# Install pnpm
corepack enable pnpm

# Install workspace deps
pnpm install

# Build the app (outputs to fluxer_app/dist/)
cd fluxer_app
pnpm build
cd ..
```

> Note: The build requires Node 22+ and pnpm. If the VM doesn't have Node installed globally,
> run the build inside a container:
> ```bash
> docker run --rm -v $(pwd):/workspace -w /workspace/fluxer_app node:24-bookworm-slim \
>   bash -lc "corepack enable pnpm && CI=true pnpm install && pnpm build"
> ```

### 4. Build the cassandra-migrate binary

The Cassandra migration runner is a Rust binary that must be pre-compiled.

```bash
# Install Rust if needed
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source ~/.cargo/env

# Build
cd scripts/cassandra-migrate
cargo build --release
cd ../..
```

The compose file expects the binary at `scripts/cassandra-migrate/target/release/cassandra-migrate`.

### 5. Set up the environment file

```bash
cp dev/.env.example dev/.env
```

Edit `dev/.env` and fill in:

| Variable | What to set |
|----------|-------------|
| `FLUXER_APP_ENDPOINT` | Your public URL e.g. `https://st.vish.gg` |
| `FLUXER_API_PUBLIC_ENDPOINT` | `https://st.vish.gg/api` |
| `FLUXER_GATEWAY_ENDPOINT` | `wss://st.vish.gg/gateway` |
| `FLUXER_MARKETING_ENDPOINT` | `https://st.vish.gg` |
| `FLUXER_ADMIN_ENDPOINT` | `https://st.vish.gg/admin` |
| `FLUXER_MEDIA_ENDPOINT` | `https://st.vish.gg/media` |
| `FLUXER_INVITE_ENDPOINT` | `https://st.vish.gg` |
| `CLOUDFLARE_TUNNEL_TOKEN` | Token from Cloudflare Zero Trust dashboard |
| `SECRET_KEY_BASE` | Generate: `openssl rand -hex 64` |
| `GATEWAY_RPC_SECRET` | Generate: `openssl rand -hex 32` |
| `GATEWAY_ADMIN_SECRET` | Generate: `openssl rand -hex 32` |
| `MEDIA_PROXY_SECRET_KEY` | Generate: `openssl rand -hex 32` |
| `SUDO_MODE_SECRET` | Generate: `openssl rand -hex 32` |
| `LIVEKIT_API_KEY` | Generate: `openssl rand -hex 16` |
| `LIVEKIT_API_SECRET` | Generate: `openssl rand -hex 32` |
| `VAPID_PUBLIC_KEY` | Generate with web-push: `npx web-push generate-vapid-keys` |
| `VAPID_PRIVATE_KEY` | (paired with above) |
| `ADMIN_OAUTH2_CLIENT_ID` | OAuth2 app ID (from Fluxer admin panel after first boot) |
| `ADMIN_OAUTH2_CLIENT_SECRET` | OAuth2 secret |
| `PASSKEY_RP_ID` | Your domain e.g. `st.vish.gg` |
| `PASSKEY_ALLOWED_ORIGINS` | `https://st.vish.gg` |

Everything else can stay at its default dev value on first boot.

### 6. Set up the LiveKit config

```bash
cp dev/livekit.yaml dev/livekit.yaml
```

Edit `dev/livekit.yaml` — replace the API key/secret with the values you set in `.env`:

```yaml
keys:
  "<LIVEKIT_API_KEY>": "<LIVEKIT_API_SECRET>"

webhook:
  api_key: "<LIVEKIT_API_KEY>"
  urls:
    - "http://api:8080/webhooks/livekit"
```

### 7. Start the stack

```bash
cd /root/fluxer/dev
docker compose up -d
```

On first boot, `cassandra-migrate` will run automatically and apply schema migrations.
ScyllaDB takes ~90 seconds to become healthy before migrations run.

Watch progress:
```bash
docker compose logs -f cassandra-migrate
docker compose logs -f api
```

### 8. Verify everything is up

```bash
docker compose ps
```

Expected running containers: `caddy`, `cloudflared`, `api`, `worker`, `gateway`, `media`, `admin`,
`marketing`, `docs`, `metrics`, `clamav`, `meilisearch`, `redis`, `minio`, `livekit`

Expected exited (one-shot): `cassandra-migrate`, `minio-setup`

The app should be accessible at your configured domain via the Cloudflare tunnel.

---

## Managing the stack

### Start / stop

```bash
cd /root/fluxer/dev

# Start all
docker compose up -d

# Stop all (keep volumes)
docker compose down

# Stop and wipe all data
docker compose down -v
```

### View logs

```bash
# All services
docker compose logs -f

# Specific service
docker compose logs -f api
docker compose logs -f gateway
```

### Rebuild a specific service after code changes

```bash
# Rebuild and restart one service
docker compose up -d --build media

# Rebuild frontend app and restart caddy
cd /root/fluxer
pnpm --filter fluxer_app build
docker compose -f dev/compose.yaml restart caddy
```

### Update from upstream

```bash
cd /root/fluxer
git pull origin main

# Rebuild frontend
pnpm install
pnpm --filter fluxer_app build

# Restart services that auto-install deps (api, worker, docs)
docker compose -f dev/compose.yaml restart api worker docs

# Rebuild services with Dockerfiles
docker compose -f dev/compose.yaml up -d --build media admin marketing metrics
```

---

## Configuration reference

### Key files

| File | Purpose |
|------|---------|
| `dev/compose.yaml` | Main docker compose for the dev stack |
| `dev/.env` | All environment variables (not committed — see `.env.example`) |
| `dev/Caddyfile.dev` | Caddy reverse proxy routing config |
| `dev/livekit.yaml` | LiveKit server config (API keys, webhook URL) |

### Port mappings (host)

| Port | Service |
|------|---------|
| `8088` | Caddy (all traffic enters here, then routed internally) |
| `7880` | LiveKit HTTP |
| `7882/udp` | LiveKit RTC |
| `7999/udp` | LiveKit RTC |
| `9042` | ScyllaDB CQL (direct access) |
| `8123` | ClickHouse (only with `--profile clickhouse`) |

All other services communicate internally over `fluxer-shared` Docker network.

### Optional profiles

```bash
# Enable ClickHouse-backed metrics
docker compose --profile clickhouse up -d

# Enable search (meilisearch is included by default in dev)
docker compose --profile search up -d
```

---

## Cloudflare tunnel setup

1. Go to Cloudflare Zero Trust → Networks → Tunnels → Create tunnel
2. Name it (e.g. `fluxer-seattle`)
3. Copy the tunnel token → set as `CLOUDFLARE_TUNNEL_TOKEN` in `dev/.env`
4. Add a public hostname:
   - Subdomain: `st` (or your chosen subdomain)
   - Domain: `vish.gg`
   - Service: `http://localhost:8088`
5. The `cloudflared` container will connect automatically on `docker compose up`

---

## Troubleshooting

### Gateway won't start

The Erlang gateway compiles from source on first start (`rebar3 compile`) — this takes 2-5 minutes.
Watch with `docker compose logs -f gateway`. If it exits, check for missing deps or compilation errors.

### ScyllaDB/Cassandra migration fails

ScyllaDB needs ~90 seconds to fully initialise. The `cassandra-migrate` container has a `sleep 30`
but on slow machines it may still fail. Restart it manually:

```bash
docker compose run --rm cassandra-migrate
```

### API exits immediately

Almost always a missing or malformed `.env` value. Check:

```bash
docker compose logs api | head -30
```

Common causes: `DATABASE_URL` unreachable, missing `SECRET_KEY_BASE`, malformed `REDIS_URL`.

### Frontend shows blank page

The `fluxer_app/dist/` directory is empty — the app hasn't been built yet.
Run the build step (Step 3 above) then restart Caddy:

```bash
docker compose restart caddy
```

### ClamAV takes forever to start

ClamAV downloads its virus definition database on first start (~500MB). This is normal.
It won't be marked healthy until the download completes (~5-10 min on first boot).
ClamAV can be disabled by setting `CLAMAV_ENABLED=false` in `.env` — the other services don't depend on it.

---

## Seattle reference deployment

The running instance at `st.vish.gg` uses this exact setup. Key details:

| Property | Value |
|----------|-------|
| Host | Seattle VM (`66.94.122.170` / Tailscale `100.82.197.124`) |
| SSH | `ssh root@seattle` or `ssh root@seattle-tailscale` |
| Install path | `/root/fluxer/` |
| Compose path | `/root/fluxer/dev/` |
| Domain | `st.vish.gg` |
| Tunnel | Cloudflare Zero Trust (`cloudflared`) |
| `AUTO_JOIN_INVITE_CODE` | `FRIENDS` (anyone with this code can register) |

To check status on seattle:

```bash
ssh seattle-tailscale "cd /root/fluxer/dev && docker compose ps"
ssh seattle-tailscale "cd /root/fluxer/dev && docker compose logs --tail 20 api"
```