homelab-optimized/docs/services/individual/crowdsec.md

# CrowdSec

**Collaborative Intrusion Detection & Prevention**

## Service Overview

| Property | Value |
|----------|-------|
| **Service Name** | crowdsec |
| **Host** | matrix-ubuntu (co-located with NPM) |
| **Category** | Security |
| **Docker Image** | `crowdsecurity/crowdsec:latest` |
| **Bouncer** | `crowdsec-firewall-bouncer-nftables` (host package) |
| **Compose File** | `hosts/vms/matrix-ubuntu/crowdsec.yaml` |
| **LAPI Port** | 8580 |
| **Metrics Port** | 6060 |

## Purpose

CrowdSec is a collaborative intrusion detection and prevention system. It analyzes logs from services (primarily NPM), detects attack patterns (brute force, scanning, CVE exploits), and blocks malicious IPs at the network layer via nftables. It shares threat intelligence with the CrowdSec community network, so your homelab benefits from crowdsourced blocklists.

## Architecture

```
Internet
    │
    ▼
nftables (crowdsec-blacklists) ── DROP banned IPs before they reach any service
    │
    ▼ (clean traffic only)
NPM (matrix-ubuntu:80/443)
    │
    └── Access logs (/opt/npm/data/logs — direct mount)
             │
             ▼
        CrowdSec Engine (Docker, localhost:8580)
             ├── Parses NPM access/error logs (all 36 proxy hosts)
             ├── Parses host syslog + auth.log
             ├── Applies scenarios (brute force, scans, CVEs)
             ├── Pushes ban decisions to firewall bouncer
             ├── Shares signals with CrowdSec community network
             └── Exposes Prometheus metrics (:6060)
                      │
                      ▼
        Firewall Bouncer (host systemd service)
             └── Syncs decisions → nftables blacklist (10s interval)
```

**Why nftables instead of nginx forward-auth?**
Some NPM proxy hosts already use `auth_request` for Authentik SSO. Nginx only allows one `auth_request` per server block, so a CrowdSec `auth_request` would conflict. The nftables approach blocks at the network layer — before packets even reach nginx — and protects all services on the host, not just NPM.

## Setup

### 1. Pre-deployment

```bash
sudo mkdir -p /opt/crowdsec/{config,data}
```

### 2. Deploy CrowdSec Engine

```bash
sudo docker compose -f /opt/homelab/hosts/vms/matrix-ubuntu/crowdsec.yaml up -d
```

### 3. Configure Log Acquisition

Create `/opt/crowdsec/config/acquis.yaml`:

```yaml
# NPM proxy host access logs
filenames:
  - /var/log/npm/proxy-host-*_access.log
labels:
  type: nginx-proxy-manager
---
# NPM proxy host error logs
filenames:
  - /var/log/npm/proxy-host-*_error.log
labels:
  type: nginx-proxy-manager
---
# Host syslog
filenames:
  - /var/log/host/syslog
  - /var/log/host/auth.log
labels:
  type: syslog
```

Restart CrowdSec after creating acquis.yaml:
```bash
sudo docker restart crowdsec
```

### 4. Install Firewall Bouncer

```bash
curl -s https://install.crowdsec.net | sudo sh
sudo apt install crowdsec-firewall-bouncer-nftables
```

### 5. Generate Bouncer API Key

```bash
sudo docker exec crowdsec cscli bouncers add firewall-bouncer
```

### 6. Configure Bouncer

Edit `/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml`:

```yaml
api_url: http://127.0.0.1:8580/
api_key: <generated-key>
deny_log: true        # log blocked packets for verification
deny_action: DROP
update_frequency: 10s
```

### 7. Start Bouncer

```bash
sudo systemctl enable --now crowdsec-firewall-bouncer
```

### 8. Enroll in CrowdSec Console (Optional)

```bash
sudo docker exec crowdsec cscli console enroll <enrollment-key>
```

Get enrollment key from https://app.crowdsec.net

## Collections

| Collection | Purpose |
|-----------|---------|
| `crowdsecurity/nginx-proxy-manager` | Parse NPM access/error logs |
| `crowdsecurity/base-http-scenarios` | HTTP brute force, path scanning, bad user agents |
| `crowdsecurity/http-cve` | Known CVE exploit detection (Log4j, etc.) |
| `crowdsecurity/linux` | SSH brute force, PAM auth failures |

## Verification

### Check nftables rules
```bash
sudo nft list set ip crowdsec crowdsec-blacklists-cscli
```

### Check bouncer status
```bash
sudo systemctl status crowdsec-firewall-bouncer
sudo docker exec crowdsec cscli bouncers list
```

### E2E test (ban → verify block → unban)
```bash
# Ban a test IP (RFC 5737 documentation range)
sudo docker exec crowdsec cscli decisions add --ip 203.0.113.50 --duration 5m --reason "e2e test"

# Wait 10-15s for bouncer sync, then verify in nftables
sudo nft list set ip crowdsec crowdsec-blacklists-cscli
# Should show: elements = { 203.0.113.50 timeout ... }

# Clean up
sudo docker exec crowdsec cscli decisions delete --ip 203.0.113.50
```

## Common Commands

```bash
# View active decisions (banned IPs)
sudo docker exec crowdsec cscli decisions list

# View alerts
sudo docker exec crowdsec cscli alerts list

# Manually ban an IP
sudo docker exec crowdsec cscli decisions add --ip 1.2.3.4 --duration 24h --reason "manual ban"

# Unban an IP
sudo docker exec crowdsec cscli decisions delete --ip 1.2.3.4

# Check installed collections
sudo docker exec crowdsec cscli collections list

# Update hub (parsers, scenarios)
sudo docker exec crowdsec cscli hub update
sudo docker exec crowdsec cscli hub upgrade

# View bouncer status
sudo docker exec crowdsec cscli bouncers list

# View metrics (log parsing, scenarios, bouncers)
sudo docker exec crowdsec cscli metrics

# Check nftables blacklist
sudo nft list set ip crowdsec crowdsec-blacklists-cscli
```

## Uptime Kuma Monitoring

- **Monitor ID:** 121
- **Group:** Matrix-Ubuntu (ID: 115)
- **Type:** HTTP
- **URL:** `http://192.168.0.154:8580/health`
- **Expected response:** `{"status":"up"}` (HTTP 200)

Note: Do NOT use `/v1/heartbeat` — it requires authentication and returns 401. The `/health` endpoint is unauthenticated.

## Deployment Status (2026-03-28)

Deployed and verified:
- CrowdSec engine parsing 16k+ log lines across all 36 NPM proxy hosts
- Firewall bouncer (nftables) active, syncing decisions every 10s
- Private IPs (192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12) auto-whitelisted
- Tailscale CGNAT range (100.64.0.0/10) whitelisted via custom local parser
- Active scenarios detecting: `http-crawl-non_statics`, `http-probing`
- E2E tested: ban → nftables blacklist → unban → cleared
- Kuma monitor active under Matrix-Ubuntu group

## Incident Log

### 2026-03-28: Tailscale client banned after PC restart
- **Affected**: shinku-ryuu (100.98.93.15) — Windows PC on Tailscale
- **Symptom**: All services behind NPM (matrix.thevish.io, etc.) unreachable from shinku-ryuu; other clients unaffected
- **Root cause**: CrowdSec banned the Tailscale IP after the PC restart generated traffic that triggered detection rules. The ban in `crowdsec-blacklists-crowdsec` nftables set dropped all packets from that IP before they reached NPM.
- **Fix**: Removed ban (`cscli decisions delete --ip 100.98.93.15`), added Tailscale CGNAT whitelist (`100.64.0.0/10`) as custom parser to prevent recurrence
- **Prevention**: The `custom/tailscale-whitelist` parser now ensures all Tailscale IPs are excluded from CrowdSec detection

## Prometheus Integration

CrowdSec exposes metrics at `http://192.168.0.154:6060/metrics`.

Add to your Prometheus config:

```yaml
  - job_name: 'crowdsec'
    static_configs:
      - targets: ['192.168.0.154:6060']
        labels:
          instance: 'matrix-ubuntu'
```

Useful metrics:
- `cs_active_decisions` — number of currently banned IPs
- `cs_alerts_total` — total alerts triggered
- `cs_parsed_total` — log lines parsed
- `cs_bucket_overflow_total` — scenario triggers

## Troubleshooting

**Legitimate traffic being blocked:**
```bash
# Check if an IP is banned
sudo docker exec crowdsec cscli decisions list --ip <ip>
# Unban if needed
sudo docker exec crowdsec cscli decisions delete --ip <ip>
```

**Whitelist your LAN and Tailscale:**

The `crowdsecurity/whitelists` parser auto-whitelists private ranges (192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12). Tailscale CGNAT IPs are whitelisted via a custom local parser:

- **File**: `/opt/crowdsec/config/parsers/s02-enrich/tailscale-whitelist.yaml`
- **Range**: `100.64.0.0/10` (Tailscale/Headscale CGNAT)
- **Verify**: `sudo docker exec crowdsec cscli parsers list | grep whitelist`

```yaml
# /opt/crowdsec/config/parsers/s02-enrich/tailscale-whitelist.yaml
name: custom/tailscale-whitelist
description: "Whitelist Tailscale/Headscale CGNAT range"
whitelist:
  reason: "tailscale CGNAT range - trusted internal traffic"
  cidr:
    - "100.64.0.0/10"
```

**Why this is critical**: CrowdSec's nftables rules run at `priority filter - 10`, **before** Tailscale's `ts-input` chain. A CrowdSec ban on a Tailscale IP blocks all traffic from that client to every service on matrix-ubuntu (NPM, Matrix, etc.), even though Tailscale would otherwise accept it. Without this whitelist, events like PC restarts can trigger false-positive bans on Tailscale clients.

**No alerts showing up:**
```bash
# Check if logs are being parsed
sudo docker exec crowdsec cscli metrics
# If parsed_total = 0, check log paths
sudo docker exec crowdsec ls -la /var/log/npm/
```

**Firewall bouncer not syncing:**
```bash
# Check bouncer service
sudo systemctl status crowdsec-firewall-bouncer
sudo journalctl -u crowdsec-firewall-bouncer -f

# Verify LAPI is responding
curl http://localhost:8580/v1/decisions

# Check bouncer registration
sudo docker exec crowdsec cscli bouncers list
```

**Bouncer config location:** `/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml`