# 🔄 Portainer Backup & Recovery Plan **Last Updated**: 2026-01-27 This document outlines the backup strategy for Portainer and all managed Docker infrastructure. --- ## Overview Portainer manages **5 endpoints** with **130+ containers** across the homelab. A comprehensive backup strategy ensures quick recovery from failures. ### Current Backup Configuration ✅ | Setting | Value | |---------|-------| | **Destination** | Backblaze B2 (`vk-portainer` bucket) | | **Schedule** | Daily at 3:00 AM | | **Retention** | 30 days (auto-delete lifecycle rule) | | **Encryption** | Yes (AES-256) | | **Backup Size** | ~30 MB per backup | | **Max Storage** | ~900 MB | | **Monthly Cost** | ~$0.005 | ### What's Backed Up | Component | Location | Backup Method | Frequency | |-----------|----------|---------------|-----------| | Portainer DB | Atlantis:/portainer | **Backblaze B2** | Daily 3AM | | Stack definitions | Git repo | Already versioned | On change | | Container volumes | Per-host | Scheduled rsync | Daily | | Secrets/Env vars | Portainer | Included in B2 backup | Daily | --- ## Portainer Server Backup ### Active Configuration: Backblaze B2 ✅ Automatic backups are configured via Portainer UI: - **Settings → Backup configuration → S3 Compatible** **Current Settings:** ``` S3 Host: https://s3.us-west-004.backblazeb2.com Bucket: vk-portainer Region: us-west-004 Schedule: 0 3 * * * (daily at 3 AM) Encryption: Enabled ``` ### Manual Backup via API ```bash # Trigger immediate backup curl -X POST "http://vishinator.synology.me:10000/api/backup/s3/execute" \ -H "X-API-Key: "REDACTED_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "accessKeyID": "004d35b7f4bf4300000000001", "secretAccessKey": "K004SyhG7s+Xv/LDB32SAJFLKhe5dj0", "region": "us-west-004", "bucketName": "vk-portainer", "password": "portainer-backup-2026", "s3CompatibleHost": "https://s3.us-west-004.backblazeb2.com" }' # Download backup locally curl -X GET "http://vishinator.synology.me:10000/api/backup" \ -H "X-API-Key: "REDACTED_API_KEY" \ -o portainer-backup-$(date +%Y%m%d).tar.gz ``` ### Option 2: Volume Backup (Manual) ```bash # On Atlantis (where Portainer runs) # Stop Portainer temporarily docker stop portainer # Backup the data volume tar -czvf /volume1/backups/portainer/portainer-$(date +%Y%m%d).tar.gz \ /volume1/docker/portainer/data # Restart Portainer docker start portainer ``` ### Option 3: Scheduled Backup Script Create `/volume1/scripts/backup-portainer.sh`: ```bash #!/bin/bash BACKUP_DIR="/volume1/backups/portainer" DATE=$(date +%Y%m%d_%H%M%S) RETENTION_DAYS=30 # Create backup directory mkdir -p $BACKUP_DIR # Backup Portainer data (hot backup - no downtime) docker run --rm \ -v portainer_data:/data \ -v $BACKUP_DIR:/backup \ alpine tar -czvf /backup/portainer-$DATE.tar.gz /data # Cleanup old backups find $BACKUP_DIR -name "portainer-*.tar.gz" -mtime +$RETENTION_DAYS -delete echo "Backup completed: portainer-$DATE.tar.gz" ``` Add to crontab: ```bash # Daily at 3 AM 0 3 * * * /volume1/scripts/backup-portainer.sh >> /var/log/portainer-backup.log 2>&1 ``` --- ## Stack Definitions Backup All stack definitions are stored in Git (git.vish.gg/Vish/homelab), providing: - ✅ Version history - ✅ Change tracking - ✅ Easy rollback - ✅ Multi-location redundancy ### Git Repository Structure ``` homelab/ ├── Atlantis/ # Atlantis stack configs ├── Calypso/ # Calypso stack configs ├── homelab_vm/ # Homelab VM configs │ ├── monitoring.yaml │ ├── openhands.yaml │ ├── ntfy.yaml │ └── prometheus_grafana_hub/ │ └── alerting/ ├── concord_nuc/ # NUC configs └── docs/ # Documentation ``` ### Backup Git Repo Locally ```bash # Clone full repo with history git clone --mirror https://git.vish.gg/Vish/homelab.git homelab-backup.git # Update existing mirror cd homelab-backup.git && git remote update ``` --- ## Container Volume Backup Strategy ### Critical Volumes to Backup | Service | Volume Path | Priority | Size | |---------|-------------|----------|------| | Grafana | /var/lib/grafana | High | ~500MB | | Prometheus | /prometheus | Medium | ~2GB | | ntfy | /var/cache/ntfy | Low | ~100MB | | Alertmanager | /alertmanager | Medium | ~50MB | ### Backup Script for Homelab VM Create `/home/homelab/scripts/backup-volumes.sh`: ```bash #!/bin/bash BACKUP_DIR="/home/homelab/backups" DATE=$(date +%Y%m%d) REMOTE="atlantis:/volume1/backups/homelab-vm" # Create local backup mkdir -p $BACKUP_DIR/$DATE # Backup critical volumes for vol in grafana prometheus alertmanager; do docker run --rm \ -v ${vol}_data:/data \ -v $BACKUP_DIR/$DATE:/backup \ alpine tar -czvf /backup/${vol}.tar.gz /data done # Sync to remote (Atlantis NAS) rsync -av --delete $BACKUP_DIR/$DATE/ $REMOTE/$DATE/ # Keep last 7 days locally find $BACKUP_DIR -maxdepth 1 -type d -mtime +7 -exec rm -rf {} \; echo "Backup completed: $DATE" ``` --- ## Disaster Recovery Procedures ### Scenario 1: Portainer Server Failure **Recovery Steps:** 1. Deploy new Portainer instance on Atlantis 2. Restore from backup 3. Re-add edge agents (they will auto-reconnect) ```bash # Deploy fresh Portainer docker run -d -p 10000:9000 -p 8000:8000 \ --name portainer --restart always \ -v /var/run/docker.sock:/var/run/docker.sock \ -v portainer_data:/data \ portainer/portainer-ee:latest # Restore from backup docker stop portainer tar -xzvf portainer-backup.tar.gz -C / docker start portainer ``` ### Scenario 2: Edge Agent Failure (e.g., Homelab VM) **Recovery Steps:** 1. Reinstall Docker on the host 2. Install Portainer agent 3. Redeploy stacks from Git ```bash # Install Portainer Edge Agent docker run -d \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /var/lib/docker/volumes:/var/lib/docker/volumes \ -v portainer_agent_data:/data \ --name portainer_edge_agent \ --restart always \ -e EDGE=1 \ -e EDGE_ID= \ -e EDGE_KEY= \ -e EDGE_INSECURE_POLL=1 \ portainer/agent:latest # Stacks will auto-deploy from Git (if AutoUpdate enabled) # Or manually trigger via Portainer API ``` ### Scenario 3: Complete Infrastructure Loss **Recovery Priority:** 1. Network (router, switch) 2. Atlantis NAS (Portainer server) 3. Git server (Gitea on Calypso) 4. Edge agents **Full Recovery Checklist:** - [ ] Restore network connectivity - [ ] Boot Atlantis, restore Portainer backup - [ ] Boot Calypso, verify Gitea accessible - [ ] Start edge agents on each host - [ ] Verify all stacks deployed from Git - [ ] Test alerting notifications - [ ] Verify monitoring dashboards --- ## Portainer API Backup Commands ### Export All Stack Definitions ```bash #!/bin/bash API_KEY=REDACTED_API_KEY BASE_URL="http://vishinator.synology.me:10000" OUTPUT_DIR="./portainer-export-$(date +%Y%m%d)" mkdir -p $OUTPUT_DIR # Get all stacks curl -s -H "X-API-Key: $API_KEY" "$BASE_URL/api/stacks" | \ jq -r '.[] | "\(.Id) \(.Name) \(.EndpointId)"' | \ while read id name endpoint; do echo "Exporting stack: $name (ID: $id)" curl -s -H "X-API-Key: $API_KEY" \ "$BASE_URL/api/stacks/$id/file" | \ jq -r '.REDACTED_APP_PASSWORD' > "$OUTPUT_DIR/${name}.yaml" done echo "Exported to $OUTPUT_DIR" ``` ### Export Endpoint Configuration ```bash curl -s -H "X-API-Key: $API_KEY" \ "$BASE_URL/api/endpoints" | jq > endpoints-backup.json ``` --- ## Automated Backup Schedule | Backup Type | Frequency | Retention | Location | |-------------|-----------|-----------|----------| | Portainer DB | Daily 3AM | 30 days | Atlantis NAS | | Git repo mirror | Daily 4AM | Unlimited | Calypso NAS | | Container volumes | Daily 5AM | 7 days local, 30 days remote | Atlantis NAS | | Full export | Weekly Sunday | 4 weeks | Off-site (optional) | --- ## Verification & Testing ### Monthly Backup Test Checklist - [ ] Verify Portainer backup file integrity - [ ] Test restore to staging environment - [ ] Verify Git repo clone works - [ ] Test volume restore for one service - [ ] Document any issues found ### Backup Monitoring Add to Prometheus alerting: ```yaml - alert: BackupFailed expr: time() - backup_last_success_timestamp > 86400 for: 1h labels: severity: warning annotations: summary: "Backup hasn't run in 24 hours" ``` --- ## Quick Reference ### Backup Locations ``` Atlantis:/volume1/backups/ ├── portainer/ # Portainer DB backups ├── homelab-vm/ # Homelab VM volume backups ├── calypso/ # Calypso volume backups └── git-mirrors/ # Git repository mirrors ``` ### Important Files - Portainer API Key: `ptr_REDACTED_PORTAINER_TOKEN` - Git repo: `https://git.vish.gg/Vish/homelab` - Edge agent keys: Stored in Portainer (Settings → Environments) ### Emergency Contacts - Synology Support: 1-425-952-7900 - Portainer Support: https://www.portainer.io/support