Sanitized mirror from private repository - 2026-04-15 05:45:58 UTC

2026-04-15 05:45:59 +00:00
commit 91841ff254
1418 changed files with 359861 additions and 0 deletions
--- a/docs/admin/b2-backup-status.md
+++ b/docs/admin/b2-backup-status.md
@@ -0,0 +1,233 @@
+# B2 Backblaze Backup Status
+
+**Last Verified**: March 21, 2026
+**B2 Endpoint**: `s3.us-west-004.backblazeb2.com`
+**B2 Credentials**: `~/.b2_env` on homelab VM
+
+---
+
+## Bucket Summary
+
+| Bucket | Host | Size | Files | Status | Lifecycle |
+|--------|------|------|-------|--------|-----------|
+| `vk-atlantis` | Atlantis (DS1823xs+) | 657 GB | 27,555 | ✅ Healthy (Hyper Backup) | Managed by Hyper Backup (smart recycle, max 30) |
+| `vk-concord-1` | Calypso (DS723+) | 937 GB | 36,954 | ✅ Healthy (Hyper Backup) | Managed by Hyper Backup (smart recycle, max 7) |
+| `vk-setillo` | Setillo (DS223j) | 428 GB | 18,475 | ✅ Healthy (Hyper Backup) | Managed by Hyper Backup (smart recycle, max 30) |
+| `vk-portainer` | Portainer (homelab VM) | 8 GB | 30 | ✅ Active | Hide after 30d, delete after 31d |
+| `vk-guava` | Guava (TrueNAS) | ~159 GB | ~3,400 | ✅ Active (Restic) | Managed by restic forget (7d/4w/3m) |
+| `vk-mattermost` | Mattermost | ~0 GB | 4 | ❌ Essentially empty | None |
+| `vk-games` | Games | 0 GB | 0 | ⚠️ Empty, **public bucket** | Delete hidden after 1d |
+| `b2-snapshots-*` | B2 internal | — | — | System bucket | None |
+
+**Estimated monthly cost**: ~$10.50/mo (at $5/TB/mo)
+
+---
+
+## Hyper Backup Configurations (per host)
+
+### Atlantis (DS1823xs+)
+
+**Hyper Backup task** → bucket `vk-atlantis`:
+- **Rotation**: Smart Recycle — daily for 7 days, weekly for 4 weeks, monthly for 3 months (max 30 versions)
+- **Encryption**: Yes (client-side)
+- **Backed up folders**:
+  - `/archive` (volume1) — long-term archival
+  - `/documents/msi_uqiyoe` (volume1) — MSI PC sync documents
+  - `/documents/pc_sync_documents` (volume1) — PC sync documents
+  - `/downloads` (volume1) — download staging
+  - `/photo` (volume2) — Synology Photos library
+  - `/homes/vish/Photos` (volume1) — user photo library
+- **Backed up apps**: CMS, FileStation, HyperBackup, OAuthService, SynologyApplicationService, SynologyDrive, SynologyPhotos, SynoFinder
+
+### Calypso (DS723+)
+
+**Hyper Backup task** → bucket `vk-concord-1`:
+- **Rotation**: Smart Recycle (max 7 versions)
+- **Encryption**: Yes (client-side)
+- **Backed up folders**:
+  - `/docker/authentik` — SSO provider data (critical)
+  - `/docker/gitea` — Git hosting data (critical)
+  - `/docker/headscale` — VPN control plane (critical)
+  - `/docker/immich` — Photo management DB
+  - `/docker/nginx-proxy-manager` — old NPM config
+  - `/docker/paperlessngx` — Document management DB
+  - `/docker/retro_site` — Personal website
+  - `/docker/seafile` — File storage data
+  - `/data/media/misc` — miscellaneous media
+  - `/data/media/music` — music library
+  - `/data/media/photos` — photo library
+- **Backed up apps**: CMS, CloudSync, DownloadStation, FileStation, GlacierBackup, HyperBackup, MariaDB10, OAuthService, StorageAnalyzer, SynologyApplicationService, SynologyPhotos, SynoFinder
+
+### Setillo (DS223j) — Tucson, AZ
+
+**Hyper Backup task** → bucket `vk-setillo`:
+- **Rotation**: Smart Recycle — daily for 7 days, weekly for 4 weeks, monthly for 3 months (max 30 versions)
+- **Encryption**: No (transit encryption only — **consider enabling data encryption**)
+- **Backed up folders**:
+  - `/backups` — backup destination
+  - `/homes/Setillo/Documents` — Edgar's documents
+  - `/homes/vish` — vish home directory
+  - `/PlexMediaServer/2015_2016_crista_green_iphone_5c` — legacy phone photos
+  - `/PlexMediaServer/other` — other media
+  - `/PlexMediaServer/photos` — photos
+- **Backed up apps**: DownloadStation, FileStation, HyperBackup, OAuthService, StorageAnalyzer, SurveillanceStation, SynoFinder, WebDAVServer
+
+---
+
+## Guava Restic Backup (vk-guava)
+
+**Tool**: Restic 0.16.4 + Rclone → Backblaze B2
+**Schedule**: Daily at 03:00 (TrueNAS cron job ID 1)
+**Encryption**: AES-256 (restic client-side, password in `/root/.restic-password`)
+**Rclone config**: `/root/.config/rclone/rclone.conf`
+**Retention**: `--keep-daily 7 --keep-weekly 4 --keep-monthly 3 --prune`
+
+**Backed up datasets:**
+| Dataset | Size | Priority |
+|---------|------|----------|
+| `/mnt/data/photos` | 158 GB | Critical |
+| `/mnt/data/cocalc` | 323 MB | Medium |
+| `/mnt/data/medical` | 14 MB | Critical |
+| `/mnt/data/website` | 58 MB | Medium |
+| `/mnt/data/openproject` | 13 MB | Medium |
+| `/mnt/data/fasten` | 5 MB | Medium |
+
+**Also backed up (added later):**
+- `/mnt/data/fenrus` (3.5 MB) — dashboard config
+- `/mnt/data/passionfruit` (256 KB) — app data
+
+**Not backed up (re-downloadable):**
+- `/mnt/data/jellyfin` (203 GB), `/mnt/data/llama` (64 GB), `/mnt/data/iso` (556 MB)
+
+**Not yet backed up (manual add):**
+- `/mnt/data/guava_turquoise` (3 TB) — see instructions below
+
+**Manual commands:**
+```bash
+# Backup
+sudo restic -r rclone:b2:vk-guava/restic \
+  --password-file /root/.restic-password \
+  backup /mnt/data/photos /mnt/data/cocalc /mnt/data/medical \
+         /mnt/data/website /mnt/data/openproject /mnt/data/fasten
+
+# List snapshots
+sudo restic -r rclone:b2:vk-guava/restic \
+  --password-file /root/.restic-password snapshots
+
+# Verify integrity
+sudo restic -r rclone:b2:vk-guava/restic \
+  --password-file /root/.restic-password check
+
+# Restore (full)
+sudo restic -r rclone:b2:vk-guava/restic \
+  --password-file /root/.restic-password \
+  restore latest --target /mnt/data/restore
+
+# Restore specific path
+sudo restic -r rclone:b2:vk-guava/restic \
+  --password-file /root/.restic-password \
+  restore latest --target /tmp/restore --include "/mnt/data/medical"
+
+# Prune old snapshots
+sudo restic -r rclone:b2:vk-guava/restic \
+  --password-file /root/.restic-password \
+  forget --keep-daily 7 --keep-weekly 4 --keep-monthly 3 --prune
+```
+
+### Adding guava_turquoise to the backup
+
+From a `root@guava` shell, follow these steps to add `/mnt/data/guava_turquoise` (3 TB) to the existing B2 backup.
+
+**1. Run a one-time backup of guava_turquoise (initial upload ~25 hrs at 30 MB/s):**
+
+```bash
+restic -r rclone:b2:vk-guava/restic \
+  --password-file /root/.restic-password \
+  -o rclone.args="serve restic --stdio --b2-hard-delete --transfers 16" \
+  backup /mnt/data/guava_turquoise
+```
+
+**2. Verify the snapshot was created:**
+
+```bash
+restic -r rclone:b2:vk-guava/restic \
+  --password-file /root/.restic-password \
+  snapshots
+```
+
+**3. Update the daily cron job to include guava_turquoise going forward:**
+
+```bash
+midclt call cronjob.query
+```
+
+Find the cron job ID (currently 1), then update it:
+
+```bash
+midclt call cronjob.update 1 '{
+  "command": "restic -r rclone:b2:vk-guava/restic --password-file /root/.restic-password -o rclone.args=\"serve restic --stdio --b2-hard-delete --transfers 16\" backup /mnt/data/photos /mnt/data/cocalc /mnt/data/medical /mnt/data/website /mnt/data/openproject /mnt/data/fasten /mnt/data/fenrus /mnt/data/passionfruit /mnt/data/guava_turquoise && restic -r rclone:b2:vk-guava/restic --password-file /root/.restic-password -o rclone.args=\"serve restic --stdio --b2-hard-delete --transfers 16\" forget --keep-daily 7 --keep-weekly 4 --keep-monthly 3 --prune"
+}'
+```
+
+**4. Verify the cron job was updated:**
+
+```bash
+midclt call cronjob.query
+```
+
+**5. (Optional) Trigger the cron job immediately instead of waiting for 3 AM:**
+
+```bash
+midclt call cronjob.run 1
+```
+
+**Cost impact:** guava_turquoise adds ~$15/mo to B2 storage (at $5/TB). After the initial upload, daily incrementals will only upload changes.
+
+---
+
+## Portainer Backup (vk-portainer)
+
+Automated daily backups of all Portainer stack configurations:
+- **Format**: Encrypted `.tar.gz` archives
+- **Retention**: Hide after 30 days, delete after 31 days
+- **Source**: Portainer backup API on homelab VM
+- **Destination**: `vk-portainer` bucket
+
+---
+
+## Checking Bucket Status
+
+```bash
+# Via B2 native API
+curl -s -u "$B2_KEY_ID:$B2_APP_KEY" \
+  https://api.backblazeb2.com/b2api/v3/b2_authorize_account
+
+# Via AWS CLI (S3-compatible)
+source ~/.b2_env
+aws s3 ls --endpoint-url https://s3.us-west-004.backblazeb2.com
+aws s3 ls s3://vk-atlantis/ --endpoint-url https://s3.us-west-004.backblazeb2.com --recursive | sort | tail -20
+```
+
+---
+
+## Rotation Policy Changes (2026-03-21)
+
+| Host | Before | After |
+|------|--------|-------|
+| **Atlantis** | rotate_earliest, max 256 versions | Smart Recycle, max 30 versions |
+| **Setillo** | rotate_earliest, max 256 versions | Smart Recycle, max 30 versions |
+| **Calypso** | Smart Recycle, max 7 versions | No change |
+
+Old versions will be pruned automatically by Hyper Backup on next scheduled run.
+
+---
+
+## Notes
+
+- All active buckets use `us-west-004` region (Backblaze B2)
+- Hyper Backup on Synology hosts handles encryption before upload
+- Guava uses restic (AES-256 encryption) — password stored in `/root/.restic-password`
+- `vk-games` is a **public** bucket — consider making it private or deleting if unused
+- `vk-setillo` has **no data encryption** — only transit encryption
+- B2 API key is stored in `~/.b2_env` and is compatible with AWS CLI S3 API
+- The `sanitize.py` script redacts B2 credentials before public repo mirroring