homelab-optimized/docs/services/individual/crowdsec.md

CrowdSec

Collaborative Intrusion Detection & Prevention

Service Overview

Property	Value
Service Name	crowdsec
Host	matrix-ubuntu (co-located with NPM)
Category	Security
Docker Image	crowdsecurity/crowdsec:latest
Bouncer	crowdsec-firewall-bouncer-nftables (host package)
Compose File	hosts/vms/matrix-ubuntu/crowdsec.yaml
LAPI Port	8580
Metrics Port	6060

Purpose

CrowdSec is a collaborative intrusion detection and prevention system. It analyzes logs from services (primarily NPM), detects attack patterns (brute force, scanning, CVE exploits), and blocks malicious IPs at the network layer via nftables. It shares threat intelligence with the CrowdSec community network, so your homelab benefits from crowdsourced blocklists.

Architecture

Internet
    │
    ▼
nftables (crowdsec-blacklists) ── DROP banned IPs before they reach any service
    │
    ▼ (clean traffic only)
NPM (matrix-ubuntu:80/443)
    │
    └── Access logs (/opt/npm/data/logs — direct mount)
             │
             ▼
        CrowdSec Engine (Docker, localhost:8580)
             ├── Parses NPM access/error logs (all 36 proxy hosts)
             ├── Parses host syslog + auth.log
             ├── Applies scenarios (brute force, scans, CVEs)
             ├── Pushes ban decisions to firewall bouncer
             ├── Shares signals with CrowdSec community network
             └── Exposes Prometheus metrics (:6060)
                      │
                      ▼
        Firewall Bouncer (host systemd service)
             └── Syncs decisions → nftables blacklist (10s interval)

Why nftables instead of nginx forward-auth? Some NPM proxy hosts already use auth_request for Authentik SSO. Nginx only allows one auth_request per server block, so a CrowdSec auth_request would conflict. The nftables approach blocks at the network layer — before packets even reach nginx — and protects all services on the host, not just NPM.

Setup

1. Pre-deployment

sudo mkdir -p /opt/crowdsec/{config,data}

2. Deploy CrowdSec Engine

sudo docker compose -f /opt/homelab/hosts/vms/matrix-ubuntu/crowdsec.yaml up -d

3. Configure Log Acquisition

Create /opt/crowdsec/config/acquis.yaml:

# NPM proxy host access logs
filenames:
  - /var/log/npm/proxy-host-*_access.log
labels:
  type: nginx-proxy-manager
---
# NPM proxy host error logs
filenames:
  - /var/log/npm/proxy-host-*_error.log
labels:
  type: nginx-proxy-manager
---
# Host syslog
filenames:
  - /var/log/host/syslog
  - /var/log/host/auth.log
labels:
  type: syslog

Restart CrowdSec after creating acquis.yaml:

sudo docker restart crowdsec

4. Install Firewall Bouncer

curl -s https://install.crowdsec.net | sudo sh
sudo apt install crowdsec-firewall-bouncer-nftables

5. Generate Bouncer API Key

sudo docker exec crowdsec cscli bouncers add firewall-bouncer

6. Configure Bouncer

Edit /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml:

api_url: http://127.0.0.1:8580/
api_key: <generated-key>
deny_log: true        # log blocked packets for verification
deny_action: DROP
update_frequency: 10s

7. Start Bouncer

sudo systemctl enable --now crowdsec-firewall-bouncer

8. Enroll in CrowdSec Console (Optional)

sudo docker exec crowdsec cscli console enroll <enrollment-key>

Get enrollment key from https://app.crowdsec.net

Collections

Collection	Purpose
crowdsecurity/nginx-proxy-manager	Parse NPM access/error logs
crowdsecurity/base-http-scenarios	HTTP brute force, path scanning, bad user agents
crowdsecurity/http-cve	Known CVE exploit detection (Log4j, etc.)
crowdsecurity/linux	SSH brute force, PAM auth failures

Verification

Check nftables rules

sudo nft list set ip crowdsec crowdsec-blacklists-cscli

Check bouncer status

sudo systemctl status crowdsec-firewall-bouncer
sudo docker exec crowdsec cscli bouncers list

E2E test (ban → verify block → unban)

# Ban a test IP (RFC 5737 documentation range)
sudo docker exec crowdsec cscli decisions add --ip 203.0.113.50 --duration 5m --reason "e2e test"

# Wait 10-15s for bouncer sync, then verify in nftables
sudo nft list set ip crowdsec crowdsec-blacklists-cscli
# Should show: elements = { 203.0.113.50 timeout ... }

# Clean up
sudo docker exec crowdsec cscli decisions delete --ip 203.0.113.50

Common Commands

# View active decisions (banned IPs)
sudo docker exec crowdsec cscli decisions list

# View alerts
sudo docker exec crowdsec cscli alerts list

# Manually ban an IP
sudo docker exec crowdsec cscli decisions add --ip 1.2.3.4 --duration 24h --reason "manual ban"

# Unban an IP
sudo docker exec crowdsec cscli decisions delete --ip 1.2.3.4

# Check installed collections
sudo docker exec crowdsec cscli collections list

# Update hub (parsers, scenarios)
sudo docker exec crowdsec cscli hub update
sudo docker exec crowdsec cscli hub upgrade

# View bouncer status
sudo docker exec crowdsec cscli bouncers list

# View metrics (log parsing, scenarios, bouncers)
sudo docker exec crowdsec cscli metrics

# Check nftables blacklist
sudo nft list set ip crowdsec crowdsec-blacklists-cscli

Uptime Kuma Monitoring

• Monitor ID: 121
• Group: Matrix-Ubuntu (ID: 115)
• Type: HTTP
• URL: http://192.168.0.154:8580/health
• Expected response: {"status":"up"} (HTTP 200)

Note: Do NOT use /v1/heartbeat — it requires authentication and returns 401. The /health endpoint is unauthenticated.

Deployment Status (2026-03-28)

Deployed and verified:

• CrowdSec engine parsing 16k+ log lines across all 36 NPM proxy hosts
• Firewall bouncer (nftables) active, syncing decisions every 10s
• Private IPs (192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12) auto-whitelisted
• Tailscale CGNAT range (100.64.0.0/10) whitelisted via custom local parser
• Active scenarios detecting: http-crawl-non_statics, http-probing
• E2E tested: ban → nftables blacklist → unban → cleared
• Kuma monitor active under Matrix-Ubuntu group

Incident Log

2026-03-28: Tailscale client banned after PC restart

• Affected: shinku-ryuu (100.98.93.15) — Windows PC on Tailscale
• Symptom: All services behind NPM (matrix.thevish.io, etc.) unreachable from shinku-ryuu; other clients unaffected
• Root cause: CrowdSec banned the Tailscale IP after the PC restart generated traffic that triggered detection rules. The ban in crowdsec-blacklists-crowdsec nftables set dropped all packets from that IP before they reached NPM.
• Fix: Removed ban (cscli decisions delete --ip 100.98.93.15), added Tailscale CGNAT whitelist (100.64.0.0/10) as custom parser to prevent recurrence
• Prevention: The custom/tailscale-whitelist parser now ensures all Tailscale IPs are excluded from CrowdSec detection

Prometheus Integration

CrowdSec exposes metrics at http://192.168.0.154:6060/metrics.

Add to your Prometheus config:

  - job_name: 'crowdsec'
    static_configs:
      - targets: ['192.168.0.154:6060']
        labels:
          instance: 'matrix-ubuntu'

Useful metrics:

• cs_active_decisions — number of currently banned IPs
• cs_alerts_total — total alerts triggered
• cs_parsed_total — log lines parsed
• cs_bucket_overflow_total — scenario triggers

Troubleshooting

Legitimate traffic being blocked:

# Check if an IP is banned
sudo docker exec crowdsec cscli decisions list --ip <ip>
# Unban if needed
sudo docker exec crowdsec cscli decisions delete --ip <ip>

Whitelist your LAN and Tailscale:

The crowdsecurity/whitelists parser auto-whitelists private ranges (192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12). Tailscale CGNAT IPs are whitelisted via a custom local parser:

• File: /opt/crowdsec/config/parsers/s02-enrich/tailscale-whitelist.yaml
• Range: 100.64.0.0/10 (Tailscale/Headscale CGNAT)
• Verify: sudo docker exec crowdsec cscli parsers list | grep whitelist

# /opt/crowdsec/config/parsers/s02-enrich/tailscale-whitelist.yaml
name: custom/tailscale-whitelist
description: "Whitelist Tailscale/Headscale CGNAT range"
whitelist:
  reason: "tailscale CGNAT range - trusted internal traffic"
  cidr:
    - "100.64.0.0/10"

Why this is critical: CrowdSec's nftables rules run at priority filter - 10, before Tailscale's ts-input chain. A CrowdSec ban on a Tailscale IP blocks all traffic from that client to every service on matrix-ubuntu (NPM, Matrix, etc.), even though Tailscale would otherwise accept it. Without this whitelist, events like PC restarts can trigger false-positive bans on Tailscale clients.

No alerts showing up:

# Check if logs are being parsed
sudo docker exec crowdsec cscli metrics
# If parsed_total = 0, check log paths
sudo docker exec crowdsec ls -la /var/log/npm/

Firewall bouncer not syncing:

# Check bouncer service
sudo systemctl status crowdsec-firewall-bouncer
sudo journalctl -u crowdsec-firewall-bouncer -f

# Verify LAPI is responding
curl http://localhost:8580/v1/decisions

# Check bouncer registration
sudo docker exec crowdsec cscli bouncers list

Bouncer config location: /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml