# 💾 Backup Strategies Guide

## Overview

This guide covers comprehensive backup strategies for the homelab, implementing the 3-2-1 backup rule and ensuring data safety across all systems.

---

## 🎯 The 3-2-1 Backup Rule

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                         3-2-1 BACKUP STRATEGY                                │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│   3 COPIES          2 DIFFERENT MEDIA       1 OFF-SITE                      │
│   ─────────         ─────────────────       ──────────                      │
│                                                                              │
│   ┌─────────┐       ┌─────────┐             ┌─────────┐                     │
│   │ Primary │       │   NAS   │             │ Tucson  │                     │
│   │  Data   │       │  (HDD)  │             │ (Remote)│                     │
│   └─────────┘       └─────────┘             └─────────┘                     │
│        +                 +                                                   │
│   ┌─────────┐       ┌─────────┐                                             │
│   │ Local   │       │  Cloud  │                                             │
│   │ Backup  │       │ (B2/S3) │                                             │
│   └─────────┘       └─────────┘                                             │
│        +                                                                     │
│   ┌─────────┐                                                               │
│   │ Remote  │                                                               │
│   │ Backup  │                                                               │
│   └─────────┘                                                               │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘
```

---

## 📊 Backup Architecture

### Current Implementation

| Data Type | Primary | Local Backup | Remote Backup | Cloud |
|-----------|---------|--------------|---------------|-------|
| Media (Movies/TV) | Atlantis | - | Setillo (partial) | - |
| Photos (Immich) | Atlantis | Calypso | Setillo | B2 (future) |
| Documents (Paperless) | Atlantis | Calypso | Setillo | B2 (future) |
| Docker Configs | Atlantis/Calypso | Syncthing | Setillo | Git |
| Databases | Various hosts | Daily dumps | Setillo | - |
| Passwords (Vaultwarden) | Atlantis | Calypso | Setillo | Export file |

---

## 🗄️ Synology Hyper Backup

### Setup Local Backup (Atlantis → Calypso)

```bash
# On Atlantis DSM:
# 1. Open Hyper Backup
# 2. Create new backup task
# 3. Select "Remote NAS device" as destination
# 4. Configure:
#    - Destination: Calypso
#    - Shared Folder: /backups/atlantis
#    - Encryption: Enabled (AES-256)
```

### Hyper Backup Configuration

```yaml
# Recommended settings for homelab backup
backup_task:
  name: "Atlantis-to-Calypso"
  source_folders:
    - /docker           # All container data
    - /photos           # Immich photos
    - /documents        # Paperless documents
  
  exclude_patterns:
    - "*.tmp"
    - "*.log"
    - "**/cache/**"
    - "**/transcode/**"    # Plex transcode files
    - "**/thumbs/**"       # Regeneratable thumbnails
  
  schedule:
    type: daily
    time: "03:00"
    retention:
      daily: 7
      weekly: 4
      monthly: 6
  
  options:
    compression: true
    encryption: true
    client_side_encryption: true
    integrity_check: weekly
```

### Remote Backup (Atlantis → Setillo)

```yaml
# For off-site backup to Tucson
backup_task:
  name: "Atlantis-to-Setillo"
  destination:
    type: rsync
    host: setillo.tailnet
    path: /volume1/backups/atlantis
  
  source_folders:
    - /docker
    - /photos
    - /documents
  
  schedule:
    type: weekly
    day: sunday
    time: "02:00"
  
  bandwidth_limit: 50 Mbps  # Don't saturate WAN
```

---

## 🔄 Syncthing Real-Time Sync

### Configuration for Critical Data

```xml
<!-- syncthing/config.xml -->
<folder id="docker-configs" label="Docker Configs" path="/volume1/docker">
    <device id="ATLANTIS-ID"/>
    <device id="CALYPSO-ID"/>
    <device id="SETILLO-ID"/>
    
    <minDiskFree unit="%">5</minDiskFree>
    <versioning type="staggered">
        <param key="maxAge" val="2592000"/>  <!-- 30 days -->
        <param key="cleanInterval" val="3600"/>
    </versioning>
    
    <ignorePattern>*.tmp</ignorePattern>
    <ignorePattern>*.log</ignorePattern>
    <ignorePattern>**/cache/**</ignorePattern>
</folder>
```

### Deploy Syncthing

```yaml
# syncthing.yaml
version: "3.8"
services:
  syncthing:
    image: syncthing/syncthing:latest
    container_name: syncthing
    hostname: atlantis-sync
    environment:
      - PUID=1000
      - PGID=1000
    volumes:
      - ./syncthing/config:/var/syncthing/config
      - /volume1/docker:/data/docker
      - /volume1/documents:/data/documents
    ports:
      - "8384:8384"   # Web UI
      - "22000:22000" # TCP sync
      - "21027:21027/udp" # Discovery
    restart: unless-stopped
```

---

## 🗃️ Database Backups

### PostgreSQL Automated Backup

```bash
#!/bin/bash
# backup-postgres.sh

BACKUP_DIR="/volume1/backups/databases"
DATE=$(date +%Y%m%d_%H%M%S)
RETENTION_DAYS=14

# List of database containers to backup
DATABASES=(
    "immich-db:immich"
    "paperless-db:paperless"
    "vaultwarden-db:vaultwarden"
    "mastodon-db:mastodon_production"
)

for db_info in "${DATABASES[@]}"; do
    CONTAINER="${db_info%%:*}"
    DATABASE="${db_info##*:}"
    
    echo "Backing up $DATABASE from $CONTAINER..."
    
    docker exec "$CONTAINER" pg_dump -U postgres "$DATABASE" | \
        gzip > "$BACKUP_DIR/${DATABASE}_${DATE}.sql.gz"
    
    # Verify backup
    if [ $? -eq 0 ]; then
        echo "✓ $DATABASE backup successful"
    else
        echo "✗ $DATABASE backup FAILED"
        # Send alert
        curl -d "Database backup failed: $DATABASE" ntfy.sh/homelab-alerts
    fi
done

# Clean old backups
find "$BACKUP_DIR" -name "*.sql.gz" -mtime +$RETENTION_DAYS -delete

echo "Database backup complete"
```

### MySQL/MariaDB Backup

```bash
#!/bin/bash
# backup-mysql.sh

BACKUP_DIR="/volume1/backups/databases"
DATE=$(date +%Y%m%d_%H%M%S)

# Backup MariaDB
docker exec mariadb mysqldump -u root -p"$MYSQL_ROOT_PASSWORD" \
    --all-databases | gzip > "$BACKUP_DIR/mariadb_${DATE}.sql.gz"
```

### Schedule with Cron

```bash
# /etc/crontab or Synology Task Scheduler
# Daily at 2 AM
0 2 * * * /volume1/scripts/backup-postgres.sh >> /var/log/backup.log 2>&1

# Weekly integrity check
0 4 * * 0 /volume1/scripts/verify-backups.sh >> /var/log/backup.log 2>&1
```

---

## 🐳 Docker Volume Backups

### Backup All Named Volumes

```bash
#!/bin/bash
# backup-docker-volumes.sh

BACKUP_DIR="/volume1/backups/docker-volumes"
DATE=$(date +%Y%m%d)

# Get all named volumes
VOLUMES=$(docker volume ls -q)

for volume in $VOLUMES; do
    echo "Backing up volume: $volume"
    
    docker run --rm \
        -v "$volume":/source:ro \
        -v "$BACKUP_DIR":/backup \
        alpine tar czf "/backup/${volume}_${DATE}.tar.gz" -C /source .
done

# Clean old backups (keep 7 days)
find "$BACKUP_DIR" -name "*.tar.gz" -mtime +7 -delete
```

### Restore Docker Volume

```bash
#!/bin/bash
# restore-docker-volume.sh

VOLUME_NAME="$1"
BACKUP_FILE="$2"

# Create volume if not exists
docker volume create "$VOLUME_NAME"

# Restore from backup
docker run --rm \
    -v "$VOLUME_NAME":/target \
    -v "$(dirname "$BACKUP_FILE")":/backup:ro \
    alpine tar xzf "/backup/$(basename "$BACKUP_FILE")" -C /target
```

---

## ☁️ Cloud Backup (Backblaze B2)

### Setup with Rclone

```bash
# Install rclone
curl https://rclone.org/install.sh | sudo bash

# Configure B2
rclone config
# Choose: New remote
# Name: b2
# Type: Backblaze B2
# Account ID: <your-account-id>
# Application Key: <your-app-key>
```

### Backup Script

```bash
#!/bin/bash
# backup-to-b2.sh

BUCKET="homelab-backups"
SOURCE="/volume1/backups"

# Sync with encryption
rclone sync "$SOURCE" "b2:$BUCKET" \
    --crypt-remote="b2:$BUCKET" \
    --crypt-password="REDACTED_PASSWORD" /root/.rclone-password)" \
    --transfers=4 \
    --checkers=8 \
    --bwlimit=50M \
    --log-file=/var/log/rclone-backup.log \
    --log-level=INFO

# Verify sync
rclone check "$SOURCE" "b2:$BUCKET" --one-way
```

### Cost Estimation

```
Backblaze B2 Pricing:
- Storage: $0.005/GB/month
- Downloads: $0.01/GB (first 1GB free daily)

Example (500GB backup):
- Monthly storage: 500GB × $0.005 = $2.50/month
- Annual: $30/year

Recommended for:
- Photos (Immich): ~500GB
- Documents (Paperless): ~50GB
- Critical configs: ~10GB
```

---

## 🔐 Vaultwarden Backup

### Automated Vaultwarden Backup

```bash
#!/bin/bash
# backup-vaultwarden.sh

BACKUP_DIR="/volume1/backups/vaultwarden"
DATE=$(date +%Y%m%d_%H%M%S)
CONTAINER="vaultwarden"

# Stop container briefly for consistent backup
docker stop "$CONTAINER"

# Backup data directory
tar czf "$BACKUP_DIR/vaultwarden_${DATE}.tar.gz" \
    -C /volume1/docker/vaultwarden .

# Restart container
docker start "$CONTAINER"

# Keep only last 30 backups
ls -t "$BACKUP_DIR"/vaultwarden_*.tar.gz | tail -n +31 | xargs -r rm

# Also create encrypted export for offline access
# (Requires admin token)
curl -X POST "http://localhost:8080/admin/users/export" \
    -H "Authorization: Bearer $VAULTWARDEN_ADMIN_TOKEN" \
    -o "$BACKUP_DIR/vaultwarden_export_${DATE}.json"

# Encrypt the export
gpg --symmetric --cipher-algo AES256 \
    -o "$BACKUP_DIR/vaultwarden_export_${DATE}.json.gpg" \
    "$BACKUP_DIR/vaultwarden_export_${DATE}.json"

rm "$BACKUP_DIR/vaultwarden_export_${DATE}.json"

echo "Vaultwarden backup complete"
```

---

## 📸 Immich Photo Backup

### External Library Backup Strategy

```yaml
# Immich backup approach:
# 1. Original photos stored on Atlantis
# 2. Syncthing replicates to Calypso (real-time)
# 3. Hyper Backup to Setillo (weekly)
# 4. Optional: rclone to B2 (monthly)

backup_paths:
  originals: /volume1/photos/library
  database: /volume1/docker/immich/postgres
  thumbnails: /volume1/docker/immich/thumbs  # Can be regenerated
```

### Database-Only Backup (Fast)

```bash
#!/bin/bash
# Quick Immich database backup (without photos)
docker exec immich-db pg_dump -U postgres immich | \
    gzip > /volume1/backups/immich_db_$(date +%Y%m%d).sql.gz
```

---

## ✅ Backup Verification

### Automated Verification Script

```bash
#!/bin/bash
# verify-backups.sh

BACKUP_DIR="/volume1/backups"
ALERT_URL="ntfy.sh/homelab-alerts"
ERRORS=0

echo "=== Backup Verification Report ==="
echo "Date: $(date)"
echo ""

# Check recent backups exist
check_backup() {
    local name="$1"
    local path="$2"
    local max_age_hours="$3"
    
    if [ ! -d "$path" ]; then
        echo "✗ $name: Directory not found"
        ((ERRORS++))
        return
    fi
    
    latest=$(find "$path" -type f -name "*.gz" -o -name "*.tar.gz" | \
             xargs ls -t 2>/dev/null | head -1)
    
    if [ -z "$latest" ]; then
        echo "✗ $name: No backup files found"
        ((ERRORS++))
        return
    fi
    
    age_hours=$(( ($(date +%s) - $(stat -c %Y "$latest")) / 3600 ))
    
    if [ $age_hours -gt $max_age_hours ]; then
        echo "✗ $name: Latest backup is ${age_hours}h old (max: ${max_age_hours}h)"
        ((ERRORS++))
    else
        size=$(du -h "$latest" | cut -f1)
        echo "✓ $name: OK (${age_hours}h old, $size)"
    fi
}

# Verify each backup type
check_backup "PostgreSQL DBs" "$BACKUP_DIR/databases" 25
check_backup "Docker Volumes" "$BACKUP_DIR/docker-volumes" 25
check_backup "Vaultwarden" "$BACKUP_DIR/vaultwarden" 25
check_backup "Hyper Backup" "/volume1/backups/hyper-backup" 168  # 7 days

# Check Syncthing status
syncthing_status=$(curl -s http://localhost:8384/rest/system/status)
if echo "$syncthing_status" | grep -q '"uptime"'; then
    echo "✓ Syncthing: Running"
else
    echo "✗ Syncthing: Not responding"
    ((ERRORS++))
fi

# Check remote backup connectivity
if ping -c 3 setillo.tailnet > /dev/null 2>&1; then
    echo "✓ Remote (Setillo): Reachable"
else
    echo "✗ Remote (Setillo): Unreachable"
    ((ERRORS++))
fi

echo ""
echo "=== Summary ==="
if [ $ERRORS -eq 0 ]; then
    echo "All backup checks passed ✓"
else
    echo "$ERRORS backup check(s) FAILED ✗"
    curl -d "Backup verification failed: $ERRORS errors" "$ALERT_URL"
fi
```

### Test Restore Procedure

```bash
#!/bin/bash
# test-restore.sh - Monthly restore test

TEST_DIR="/volume1/restore-test"
mkdir -p "$TEST_DIR"

# Test PostgreSQL restore
echo "Testing PostgreSQL restore..."
LATEST_DB=$(ls -t /volume1/backups/databases/immich_*.sql.gz | head -1)
docker run --rm \
    -v "$TEST_DIR":/restore \
    -v "$LATEST_DB":/backup.sql.gz:ro \
    postgres:15 \
    bash -c "gunzip -c /backup.sql.gz | psql -U postgres"

# Verify tables exist
if docker exec test-postgres psql -U postgres -c "\dt" | grep -q "assets"; then
    echo "✓ PostgreSQL restore verified"
else
    echo "✗ PostgreSQL restore failed"
fi

# Cleanup
rm -rf "$TEST_DIR"
```

---

## 📋 Backup Schedule Summary

| Backup Type | Frequency | Retention | Destination |
|-------------|-----------|-----------|-------------|
| Database dumps | Daily 2 AM | 14 days | Atlantis → Calypso |
| Docker volumes | Daily 3 AM | 7 days | Atlantis → Calypso |
| Vaultwarden | Daily 1 AM | 30 days | Atlantis → Calypso → Setillo |
| Hyper Backup (full) | Weekly Sunday | 6 months | Atlantis → Calypso |
| Remote sync | Weekly Sunday | 3 months | Atlantis → Setillo |
| Cloud sync | Monthly | 1 year | Atlantis → B2 |
| Syncthing (configs) | Real-time | 30 days versions | All nodes |

---

## 🔗 Related Documentation

- [Disaster Recovery](../troubleshooting/disaster-recovery.md)
- [Synology Disaster Recovery](../troubleshooting/synology-disaster-recovery.md)
- [Offline Password Access](../troubleshooting/offline-password-access.md)
- [Storage Topology](../diagrams/storage-topology.md)
- [Portainer Backup](portainer-backup.md)
