homelab-optimized/docs/CHANGELOG.md

Changelog

2026-03-27

Security

• crowdsec: Deployed CrowdSec intrusion detection + prevention on matrix-ubuntu, co-located with NPM. Engine parses all 36 NPM proxy host logs + host syslog. Firewall bouncer (nftables) blocks banned IPs at the network layer — avoids nginx auth_request conflicts with Authentik SSO. Kuma monitor added (ID 121, /health endpoint). Prometheus metrics on :6060.

Monitoring

• grafana dashboards: Complete overhaul — 6 dashboards auto-provisioned from bind-mounted JSON files (/home/homelab/docker/grafana-dashboards/). Removed 900+ lines of embedded dashboard JSON from monitoring.yaml. Pinned Prometheus datasource UID (cfbskvs8upds0b).
• grafana new dashboards: Added Synology NAS Monitoring (SNMP disk temps/status, CPU, memory, volumes, network for Atlantis + Calypso), TrueNAS Guava Monitoring (CPU, RAM, ZFS pools, disk I/O), Tailscale Bandwidth (per-host TX/RX rates).
• grafana fixes: Fixed Infrastructure Overview + old Synology dashboard empty datasource UIDs. Fixed $job variable allValue (was empty string, now .*). Cleaned up duplicate provisioned synology-dashboard-v2 ghost dashboard (required Grafana volume wipe). Setillo (DS223j) now showing in Synology dashboard after restarting stopped exporters.
• kuma: Added Setillo Node Exporter (ID 122) and SNMP Exporter (ID 123) monitors under Setillo group.
• frigate: Tested Frigate NVR on Seattle with Tapo camera (192.168.68.67) via Tailscale subnet routing. CPU detection working, go2rtc restreaming confirmed. Removed after validation — docs saved for future permanent deployment.
• tailscale: Enabled --accept-routes=true on Seattle to allow access to NUC's 192.168.68.0/22 subnet. NUC route was already advertised and approved in Headscale.
• tdarr: Synced all nodes to v2.66.01 (server was 2.65.01, Calypso node was 2.64.02). Redeployed arr-stack on Atlantis, tdarr-node on Calypso, Guava, PVE LXC. Expanded PVE LXC disk 16GB→32GB (was 100% full), pruned 2.86GB old images.

Fixes

• immich (calypso): Fixed Immich-SERVER crash (getaddrinfo ENOTFOUND database). Portainer git deploy does not load env_file references — all env vars (DB_HOSTNAME, DB_PASSWORD, etc.) added as Portainer stack environment overrides via API.
• kuma: Fixed broken monitor list caused by malformed accepted_statuscodes_json field ([200-299] → ["200-299"]) in CrowdSec monitor entry. Fixed CrowdSec health check URL from /v1/heartbeat (requires auth, returns 401) to /health (unauthenticated, returns 200).

Infrastructure

• setillo: Configured vish user for docker access — added to wheel group (NOPASSWD sudo), added /usr/local/bin to PATH via .profile. Docker (Synology ContainerManager) now accessible without full path or root login.
• matrix-ubuntu: VM resized — 16GB RAM (was ~8GB), 1TB disk (was smaller). LV extended online from 97GB to 1005GB via growpart + pvresize + lvextend -r. Now 893GB free (8% used).
• mcp: Added seattle as SSH host alias in homelab MCP server (alongside existing seattle-tailscale).
• photoprism (jellyfish): Started PhotoPrism container on jellyfish (/srv/nas/ametrine/Docker/photoprism/, port 2342).

Container Inventory (2026-03-27)

Host	Running	Stopped	Total
Atlantis	59	0	59
Calypso	62	0	62
Homelab-VM	37	1	38
Concord NUC	22	0	22
Matrix-Ubuntu	12	0	12
Guava	28	6	34
Seattle	19	1	20
RPi5	7	0	7
Jellyfish	1	1	2
Total	247	9	256

2026-03-25

Infrastructure

• portainer: Updated server 2.39.0 → 2.39.1 LTS on atlantis. Updated edge agents to 2.39.1 on all 4 endpoints (homelab-vm, calypso, nuc, rpi5).
• portainer stacks: Fixed stale git credentials across atlantis and calypso. Cleaned up orphan Docker Compose projects (containers created outside Portainer with mismatched project labels) on atlantis, calypso, and homelab-vm.
• netbox: Migrated from standalone docker compose to Portainer GitOps stack (ID 738) on homelab-vm.
• semaphore: Removed — replaced by CLI + cron + MCP workflow. Compose archived.

Features

• AGENTS.md: Overhauled Vesper agent identity — structured priorities, multi-host task guidance, failure handling, context budget, known footguns, tailscale mesh runbook.
• MCP tools: Added 5 Authentik SSO tools — create_proxy_provider, create_application, list_sessions, delete_session, get_events. Service onboarding is now 2 MCP calls.
• email backup: Daily incremental backup of 3 email accounts (dvish92, lzbellina92, admin@thevish.io) to atlantis NFS mount at /volume1/archive/old_emails/. IMAP auto-reconnect on Gmail throttling. Cron at 3 AM.

Fixes

• NFS mount: Fixed atlantis /volume1/archive NFS export — removed krb5i (no Kerberos configured), added LAN routing rule to bypass Tailscale for 192.168.0.0/24.
• ansible inventory: Commented out offline hosts (pi-5-kevin, moon) to prevent exit code 4 on every playbook run.
• image update docs: Added step-by-step walkthrough, orphan container gotcha, and git auth troubleshooting.
• moon jellyfish mount: Added noserverino to CIFS mount — fixed "folder contents cannot be displayed" error in GUI file manager.
• moon guava backup: NFS mount from atlantis (100.83.230.112:/volume1/archive/guava_full_backup → /home/moon/guava_backup_atlantis), read-only over Tailscale. Added 100.64.0.6 to atlantis NFS export, persisted in fstab.
• olares investigation: Documented Olares internal Headscale/Tailscale architecture — runs its own coordination server inside k3s for reverse proxy tunneling. Cannot be replaced with external Headscale without breaking *.olares.com remote access.

Stable Diffusion Forge (shinku-ryuu)

• Forge WebUI: Installed Stable Diffusion WebUI Forge on shinku-ryuu (RTX 4080, 16GB VRAM, i7-14700K, 96GB RAM). Conda env with Python 3.10, SDXL Base 1.0 model. Access at http://100.98.93.15:7860 or http://localhost:7860. Launcher: C:\stable-diffusion-webui-forge\run-forge.bat.
• Guava Gitea: Increased avatar max file size from 1MB to 10MB in /etc/gitea/app.ini.

Git Migration

• playgrounds → Guava Gitea: Migrated 35 git repos from moon (~/Documents/playgrounds/) to Guava Gitea (http://guava.crista.home:30008) under the lulupearl user. Sources: 8 bitbucket, 26 gitlab, 1 lulupearl_gitea. All repos private, commit history preserved. Cloned all 34 repos to homelab-vm at /home/homelab/organized/repos/.

Tailscale Mesh Verification

• Verified full 30-path mesh across 6 SSH-accessible hosts. All direct connections. Setillo uses DERP initially but hole-punches to direct (~55ms WAN latency). Documented Synology-specific tailscale CLI paths and ping limitations.

[Unreleased] (2026-02-27)

Bug Fixes

• credentials: Restored all credentials broken by sanitization commit 037d766a

  • Affected stacks: authentik-sso, paperless, wireguard (calypso+nuc), monitoring, dyndns (atlantis+nuc), watchtower, yourspotify, paperless-ai, alerting
  • Root cause: sanitization commit replaced real values with REDACTED_PASSWORD placeholders across 14+ compose files; containers redeployed with broken env vars
  • Fix: recovered original values from git history (037d766a^) and pushed as commits 50d8eca8 and 4e5607b7; all 11 affected stacks redeployed via API

• portainer: Updated portainer-homelab saved Git credential with new Gitea token

  • Previously expired token caused all 43 stacks using credId:1 to fail git pulls
  • Fixed via PUT /api/users/1/gitcredentials/1

• portainer-api-guide: Corrected authentication docs — ptr_* tokens require X-API-Key header, not Authorization: Bearer; updated version 2.33.7 → 2.39.0

[Unreleased] (2025-02-12)

Features

• arr-suite: Implement Trash Guides language configuration for Radarr and Sonarr
  • Added 4 custom formats: Language Not English (-10000), Anime Dual Audio (+500), Multi (+500), Language Not Original (0)
  • Updated quality profiles to prioritize English content while allowing foreign films in original language
  • Enhanced anime support with dual audio preference
  • Enables proper handling of foreign films like "Cold War" in Polish
  • Documentation: docs/arr-suite-language-configuration.md

0.10.3 (2026-02-07)

Bug Fixes

• update Revolt -> Stoat in email titles/desc. (#508) (84483ce)

0.10.2 (2026-01-25)

Bug Fixes

• thREDACTED_APP_PASSWORD requires rgb8/rgba8 (#505) (413aa04)

0.10.1 (2026-01-25)

Bug Fixes

• use Rust 1.92.0 for Docker build (#503) (98da8a2)

0.10.0 (2026-01-25)

Features

• allow kicking members from voice channels (#495) (0dc5442)
• repository architecture for files crate w. added tests (#498) (01ded20)

Bug Fixes

• expose ratelimit headers via cors (#496) (a1a2125)

0.9.4 (2026-01-10)

Bug Fixes

• checkout repo. before bumping lock (#490) (b2da2a8)
• persist credentials for git repo (#492) (c674a9f)

0.9.3 (2026-01-10)

Bug Fixes

• pipeline fixes (#487) (aeeafeb)

0.9.2 (2026-01-10)

Bug Fixes

• disable publish for services (#485) (d13609c)

0.9.1 (2026-01-10)

Bug Fixes

• ci: pipeline fixes (marked as fix to force release) (#483) (303e52b)

0.9.0 (2026-01-10)

Features

• add id field to role (#470) (2afea56)
• add ratelimits to gifbox (1542047)
• include groups and dms in fetch mutuals (caa8607)
• include member payload in REDACTED_APP_PASSWORD event (480f210)
• initial work on tenor gif searching (b0c977b)
• make message lexer use unowned string (1561481)
• ready payload field customisation (db57706)
• require auth for search (b5cd5e3)
• trending and categories routes (5885e06)
• voice chats v2 (#414) (d567155)

Bug Fixes

• add license to revolt-parser (5335124)
• allow for disabling default features (65fbd36)
• apple music to use original url instead of metadata url (bfe4018)
• apply uname fix to january and autumn (8f9015a)
• ci: publish images under stoatchat and remove docker hub (d65c1a1)
• correct miniz_oxide in lockfile (#478) (5d27a91)
• correct shebang for try-tag-and-release (050ba16)
• correct string_cache in lockfile (#479) (0b178fc)
• don't remove timeouts when a member leaves a server (#409) (e635bc2)
• don't update the same field while trying to remove it (f4ee35f), closes #392
• github webhook incorrect payload and formatting (#468) (dc9c82a)
• implement Serialize to ClientMessage (dea0f67)
• newly created roles should be ranked the lowest (947eb15)
• permit empty remove array in edit requests (6ad3da5)
• preserve order of replies in message (#447) (657a3f0)
• prevent timing out members which have TimeoutMembers permission (e36fc97)
• relax settings name regex (3a34159)
• remove authentication tag bytes from attachment download (32e6600)
• rename openapi operation ids (6048587), closes #406
• respond with 201 if no body in requests (#465) (24fedf8)
• swap to using reqwest for query building (38dd4d1)
• use trust_cloudflare config value instead of env var (cc7a796)
• use our own result types instead of tenors types (a92152d)