# Homelab Operational Runbooks

This directory contains step-by-step operational runbooks for common homelab management tasks. Each runbook provides clear procedures, prerequisites, and rollback steps.

## 📚 Available Runbooks

### Service Management
- **[Add New Service](add-new-service.md)** - Deploy new containerized services via GitOps
- **[Service Migration](service-migration.md)** - Move services between hosts safely
- **[Add New User](add-new-user.md)** - Onboard new users with proper access

### Infrastructure Maintenance
- **[Disk Full Procedure](disk-full-procedure.md)** - Handle full disk scenarios
- **[Certificate Renewal](certificate-renewal.md)** - Manage SSL/TLS certificates
- **[Synology DSM Upgrade](synology-dsm-upgrade.md)** - Safely upgrade NAS firmware

### Security
- **[Credential Rotation](credential-rotation.md)** - Rotate exposed or compromised credentials

## 🎯 How to Use These Runbooks

### Runbook Format
Each runbook follows a standard format:
1. **Overview** - What this procedure accomplishes
2. **Prerequisites** - What you need before starting
3. **Estimated Time** - How long it typically takes
4. **Risk Level** - Low/Medium/High impact assessment
5. **Procedure** - Step-by-step instructions
6. **Verification** - How to confirm success
7. **Rollback** - How to undo if something goes wrong
8. **Troubleshooting** - Common issues and solutions

### When to Use Runbooks
- **Planned Maintenance** - Follow runbooks during scheduled maintenance windows
- **Incident Response** - Use as quick reference during outages
- **Training** - Onboard new admins with documented procedures
- **Automation** - Use as basis for creating automated scripts

### Best Practices
- ✅ Always read the entire runbook before starting
- ✅ Have a rollback plan ready
- ✅ Test in development/staging when possible
- ✅ Take snapshots/backups before major changes
- ✅ Document any deviations from the runbook
- ✅ Update runbooks when procedures change

## 🚨 Emergency Procedures

For emergency situations, refer to:
- [Emergency Access Guide](../troubleshooting/EMERGENCY_ACCESS_GUIDE.md)
- [Recovery Guide](../troubleshooting/RECOVERY_GUIDE.md)
- [Disaster Recovery](../troubleshooting/disaster-recovery.md)

## 📋 Runbook Maintenance

### Contributing
When you discover a new procedure or improvement:
1. Create a new runbook using the template below
2. Follow the standard format
3. Include real examples from your infrastructure
4. Test the procedure before documenting

### Runbook Template
```markdown
# [Procedure Name]

## Overview
Brief description of what this accomplishes and when to use it.

## Prerequisites
- [ ] Required access/credentials
- [ ] Required tools/software
- [ ] Required knowledge/skills

## Metadata
- **Estimated Time**: X minutes/hours
- **Risk Level**: Low/Medium/High
- **Requires Downtime**: Yes/No
- **Reversible**: Yes/No
- **Tested On**: Date last tested

## Procedure

### Step 1: [Action]
Detailed instructions...

```bash
# Example commands
```

Expected output:
```
Example of what you should see
```

### Step 2: [Next Action]
Continue...

## Verification
How to confirm the procedure succeeded:
- [ ] Verification step 1
- [ ] Verification step 2

## Rollback Procedure
If something goes wrong:
1. Step to undo changes
2. How to restore previous state

## Troubleshooting
**Issue**: Common problem
**Solution**: How to fix it

## Related Documentation
- [Link to related doc](path)

## Change Log
- YYYY-MM-DD - Initial creation
- YYYY-MM-DD - Updated for new procedure
```

## 📞 Getting Help

If a runbook is unclear or doesn't work as expected:
1. Check the troubleshooting section
2. Refer to related documentation links
3. Review the homelab monitoring dashboards
4. Consult the [Infrastructure Overview](../infrastructure/INFRASTRUCTURE_OVERVIEW.md)

## 📊 Runbook Status

| Runbook | Status | Last Updated | Tested On |
|---------|--------|--------------|-----------|
| Add New Service | ✅ Active | 2026-02-14 | 2026-02-14 |
| Service Migration | ✅ Active | 2026-02-14 | 2026-02-14 |
| Add New User | ✅ Active | 2026-02-14 | 2026-02-14 |
| Disk Full Procedure | ✅ Active | 2026-02-14 | 2026-02-14 |
| Certificate Renewal | ✅ Active | 2026-02-14 | 2026-02-14 |
| Synology DSM Upgrade | ✅ Active | 2026-02-14 | 2026-02-14 |
| Credential Rotation | ✅ Active | 2026-02-20 | — |

---

**Last Updated**: 2026-02-14